Disentangling generalization and memorization in large language models using chess

arXiv:2601.16823 · cs.CL, cs.AI · Submitted 2026-01-23 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Disentangling generalization and memorization in large language models using chess".

Jane: Here's the gist of the paper. The gist The results reveal a consistent performance gradient across all tested models: performance declines as the density of relevant priors decreases, highlighting persistent weaknesses in state-of-the-art LLMs when generalizing beyond the training data manifold.

Tom: Conceptual Framework The study aims to disentangle memorization and generalization by formalizing these concepts in terms of performance under different data distributions Let ptrain(x) denote the (unknown) training data distribution of an LLM and let ptest(x) be a test distribution over inputs x ∈ X within a given domain.

Jane: Memorization is defined as the model’s ability to perform accurately under WD conditions, i.e., when x ∼ ptrain(x), which reflects the model’s ability to solve tasks when relevant priors are available, predominantly through retrieval and recall of previously observed solutions.

Tom: Generalization is defined as the model’s ability to perform accurately under OOD conditions, i.e., when x ∼ ptest(x) with ptest ̸= ptrain, reflecting the model’s ability to solve tasks when relevant priors are sparse or entirely unavailable, indicating systematic and adaptive first-principle reasoning beyond previously memorized solutions to solve novel tasks within a familiar domain.

Paper summary: Tom: So we're looking at this paper today titled "Disentangling generalization and memorization in large language models using chess". The authors are Leonard Pleiss, Maximilian Schiffer, and Robert K. von Weizsäcker. The main idea they’re pushing is that we need a way to figure out if these massive language models are just really good at recalling things they saw before or if they actually have some kind of real reasoning ability when faced with something new.

Jane: That sounds like a really important question for any of us using these models every day. What do they claim is their main approach for testing this?

Tom: They introduce chess as a controlled testbed for this. Instead of just looking at text tasks, they’re building a taxonomy of chess positions based on how much relevant prior information is in the position. It ranges from states that are common and easily solved by memorization to completely novel ones that demand generalization.

Lu: What’s interesting about their setup is that they do this without needing to know the specific training data of the models being tested. They are trying to create a framework for this distinction empirically, using chess as a structured environment for studying strategic reasoning in large language models (reference: <ref:2601.16823#pg3>).

Meng: From an engineering standpoint, that sounds like a solid way to try and separate the recall from the generalization part of the AI’s performance. But I always wonder how they're going to actually define what 'relevant priors' means in chess terms without looking at the training data?

Lalam: The paper defines this distinction using two conditions. Memorization is when a model performs accurately under WD conditions, which is when the input position occurs frequently in the training data and it reflects recall of previously observed solutions.

Tom: And generalization, they define that as out-of-distribution performance, or OOD performance, where the input is sampled from a test distribution different from the training distribution. This means testing if the model can solve novel tasks when relevant prior information is sparse or unavailable (reference: <ref:2601.16823#pg2>).

Paper summary: Jane: That makes perfect sense for understanding how we assess these models in the real world. But how do they actually build these test sets? They can't just pick random positions if they want to be rigorous.

Tom: They approximate the density of relevant priors by categorizing chess positions into three subsets: Within-distribution, or WD, Near-distribution, ND, and Out-of-Distribution, or OOD. WD positions are those that occur at least one thousand times in the Lichess Masters database <ref:2601.16823#pg1>.

Lu: That’s a very concrete way to operationalize the concept of prior density. The next category is ND positions, which they generate by performing ten random legal moves from the starting position and discarding any that are already in their database, testing associative recall.

Meng: So they are creating controlled data distributions based on how common or rare a state is in existing play versus how rare it is overall. That seems like a practical way to manage the complexity of defining priors. But what about OOD positions? Are those just pure random noise?

Lalam: OOD positions are constructed by uniformly sampling legal configurations through random placement of ten pieces per side onto the board, allowing overwrites and subject to position validity constraints. These are theoretically feasible yet atypical board states that are extremely unlikely to arise during human play (reference: <ref:2601.16823#pg2>).

Jane: Okay, so they’ve got a way to generate these scenarios systematically. But what is the actual finding when they combine this framework with the GPT lineage and other modern models like Claude Opus and Gemini?

Tom: The analysis shows a steep gradient where performance consistently degrades as the density of relevant priors decreases. For tasks with few relevant priors, base model performance regresses to the random-play baseline, and all models propose illegal moves at sharply increasing rates (reference: <ref:2601.16823#pg1>).

Lu: That suggests that for novel positions, the models fail to reliably apply the fixed rules of chess in unfamiliar situations. Even when we use reasoning augmentation, like chain-of-thought inference, it improves performance across all conditions but the relative marginal benefit of that reasoning diminishes with decrease in availability of relevant priors (reference: <ref:2601.16823#pg3>).

Paper summary: Meng: So if I’m deploying this to production, this suggests that adding more reasoning steps isn't going to fix the fundamental problem when we hit truly novel inputs. It just makes the model guess more often when it doesn't have a pattern anchor.

Lalam: The results point toward limitations in systematic generalization, showing a need for mechanisms beyond just scaling model size to get robust performance when deprived of relevant priors (reference: <ref:2601.16823#pg1>).

Jane: So what does this all mean for the broader field? What is the big picture implication of this research on LLMs and AI in general?

Tom: It suggests that scaling model size alone doesn't solve the problem of generalization. The gains diminish sharply for tasks with few relevant priors, which indicates that increasing model capacity primarily enhances recall and pattern completion within data-rich regions of the training distribution (reference: <ref:2601.16823#pg3>).

Lu: It’s showing that current architectures are very good at mastering the training data manifold, but they struggle significantly when forced to reason from first principles in structurally novel spaces.

Meng: For deployment, this means we have to be extra careful about where we test our models. If we expect them to handle a lot of unseen scenarios, this framework tells us exactly where their performance will likely drop off a cliff.

Lalam: The paper highlights that memorization remains a dominant strength, but base model performance collapses when generalization is required in this context (reference: <ref:2601.16823#pg1>).

Jane: So to wrap up, the authors of "Disentangling generalization and memorization in large language models using chess" are using a structured game like chess to show that LLMs have a clear performance gradient based on how much relevant prior information is available, and that scaling them up doesn't fix the fundamental weakness when they run out of familiar patterns.

Tom: Exactly. The core finding is that we need better mechanisms beyond just making the models bigger to get robust performance in those novel areas. That’s what this paper points toward for future work on LLM development.

Conclusion: Tom: So, we’re wrapping up this discussion on "Disentangling generalization and memorization in large language models using chess." The authors are Leonard Pleiss, Maximilian Schiffer, and Robert K. von Weizsäcker.

Jane: That paper is really about figuring out if these huge language models are just great at remembering what they’ve seen or if they actually have some real ability to think for themselves when faced with something completely new.

Tom: Yeah, they use chess as this controlled testbed to separate those two things, and the core finding is that there's a consistent performance drop whenever you move from familiar positions to really novel ones.

Jane: It shows that even when these models get smarter, their ability to handle totally unfamiliar situations without having seen similar patterns is still pretty weak.

Lu: What this means for the architecture is that scaling up the model size doesn't fix the generalization problem; it just makes them better at recalling things they already know.

Meng: From a deployment standpoint, it tells us that we can’t just throw bigger models at every new situation and expect robust performance if those situations are sparse in training data.

Lalam: I'm pretty sure this is accurate because the results show that base model performance actually drops to the random play baseline when relevant prior information gets really scarce.

Tom: So, the implication is that we need better ways to teach models first principles instead of just massive data recall.

Jane: It suggests we need mechanisms beyond just making models bigger to get reliable results when they run out of familiar patterns.

Leonard S. Pleiss, Maximilian Schiffer, Robert K. von Weizsäcker

Technical University of Munich

cs.CL, cs.AI

Submitted: 2026-01-23

Updated: 2026-10-20

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: The gist The results reveal a consistent performance gradient across all tested models: performance declines as the density of relevant priors decreases, highlighting persistent weaknesses in

Key concepts

Memorization: This is the model's ability to solve tasks accurately only when the input closely matches what it has seen during training. It relies heavily on retrieving previously observed solutions rather than understanding underlying rules or first principles.
Generalization: This is the model's ability to perform well on completely new or unseen inputs, even if they are structurally different from its training data. True generalization requires applying learned knowledge to novel situations without relying on exact prior examples.
Prior Density: This measures how much relevant, known information (priors) is available for a specific input position. High density means many similar positions exist in the training data, while low density means the position is novel and lacks recognizable patterns.
Generalization Gap (Rgen): This metric quantifies the difference between how well a model performs on familiar data versus how well it performs on new, unseen data. A large gap suggests poor generalization capabilities when moving from known to unknown scenarios.

Terminology

Generalization — 3× in this paper · explained in 14 episodes
Memorization — 2× in this paper · explained in 4 episodes
Language Models (LLMs) — 1× in this paper · explained in 2 episodes
Large Language Models (LLMs) — 1× in this paper · explained in 8 episodes

Summary

The gist The results reveal a consistent performance gradient across all tested models: performance declines as the density of relevant priors decreases, highlighting persistent weaknesses in state-of-the-art LLMs when generalizing beyond the training data manifold.

Conceptual Framework

The study aims to disentangle memorization and generalization by formalizing these concepts in terms of performance under different data distributions Let ptrain(x) denote the (unknown) training data distribution of an LLM and let ptest(x) be a test distribution over inputs x ∈ X within a given domain. Memorization is defined as the model’s ability to perform accurately under WD conditions, i.e., when x ∼ ptrain(x), which reflects the model’s ability to solve tasks when relevant priors are available, predominantly through retrieval and recall of previously observed solutions. Generalization is defined as the model’s ability to perform accurately under OOD conditions, i.e., when x ∼ ptest(x) with ptest ̸= ptrain, reflecting the model’s ability to solve tasks when relevant priors are sparse or entirely unavailable, indicating systematic and adaptive first-principle reasoning beyond previously memorized solutions to solve novel tasks within a familiar domain. To operationalize this distinction, the generalization gap is quantified as Rgen = Ex∼ptest [L(f(x))] / Ex∼ptrain [L(f(x)], where L(f(x)) is a loss or performance metric associated with the model’s output f(x) on input x.

Approximating Prior Density in Chess

The density of relevant priors is approximated by categorizing chess positions into three subsets: Within-distribution (WD), Near-distribution (ND), and Out-of-distribution (OOD). WD positions are defined as those that occur at least nmin = 1,000 times in the Lichess Masters database, approximating ptrain(xWD) ≈ maxx∈X ptrain(x), which suggests models can likely achieve strong performance through pure memorization. ND positions are generated by performing k=10 random legal moves (20 plies) from the starting position and discarding any position that appears in the database, testing associative recall. XOOD positions are constructed by uniformly sampling legal configurations through random placement of ten pieces per side onto the board, allowing overwrites and subject to position validity constraints, creating theoretically feasible yet atypical board states that are extremely unlikely to arise during human play.

Performance Analysis Across Model Generations

The analysis focuses primarily on OpenAI’s GPT lineage across three generations (GPT-3.5, GPT-4o, and GPT-5) to isolate the effects of scale and reasoning augmentation. The average centipawn loss shows a consistent decrease as relevant priors become sparser. For instance, GPT-3.5 achieved an average ACPL of 807.97 (SD = 368.58), GPT-4o of 540.38 (SD = 461.25), and GPT-5 of 463.5 (SD = 463.11). Despite progress, GPT-3.5, 4o and 5 perform worse than random play across all conditions, with GPT-5 in WD positions being the sole exception.

Impact of Reasoning and Scaling

The role of chain-of-thought reasoning is examined by comparing GPT-5 with minimal reasoning effort against moderate reasoning effort. Additional reasoning effort markedly improved performance across all conditions, reducing both average centipawn loss (left panel) and frequency of illegal moves (right panel). However, the relative marginal benefit of reasoning diminishes with decrease in availability of relevant priors. In OOD positions, the ACPL was approximately 298.81 (SD = 427.61) with moderate reasoning, suggesting that even with reasoning enabled, the chain-of-thought process struggles to construct valid strategies in the absence of recognizable pattern anchors.

Conclusion and Implications

The findings reveal that memorization remains a dominant strength, but base model performance collapses when generalization is required. Scaling current architectures shows diminishing returns, as the magnitude of gains decreases sharply for tasks with few relevant priors. Reasoning primarily acts as a retrieval optimizer rather than an engine of first-principles derivation, and overall results point towards severe weaknesses with respect to LLMs’ OOD performance within the game of chess. This suggests that increasing model capacity primarily enhances recall and pattern completion within data-rich regions of the training distribution, while leaving structurally novel or low-density regions largely unresolved.

Limitations

The work acknowledges limitations, including the inability to guarantee absolute separation of WD and OOD positions within the training corpus and the potential for confounding between a position’s distribution proximity and its inherent tactical complexity. Furthermore, centipawn loss evaluations via chess engines are inherently imperfect since the game is not fully solved. The results suggest that scaling model size alone may not suffice to enable robust generalization, implying that increasing model capacity primarily enhances recall and pattern completion within data-rich regions of the training distribution.

Implications Beyond Chess

The motifs observed in this study—robust memorization, brittle generalization, and diminishing returns through scaling current architectures and training paradigms—may not be idiosyncratic to the chess domain. The findings possibly represent a manifestation of a more general phenomenon in LLMs that defies measurement in other domains: the systemic difficulty of inferring structural rules to solve novel tasks without close priors. This suggests that increasing model capacity primarily enhances recall and pattern completion within data-rich regions of the training distribution, while leaving structurally novel or low-density regions largely unresolved. The diminishing gains observed across successive model generations further indicate that scaling model size alone may not suffice to enable robust generalization.

Hardware Specification

All experiments were performed on a 2024 MacBook Air with an Apple M3 processor. The study utilized a locally deployed variant of the Lc0 network t3-512x15x16h-distill-swa-2767500, which represents a medium-sized configuration designed to balance performance and efficiency across both GPU and CPU settings. The paper utilized the python-chess library for dataset generation. The study employed Stockfish 17.1 at search depth 30 [Romstad et al., 2008], which provides a robust trade-off between computational cost and evaluation reliability. The study employed Leela Chess Zero (Lc0, Pascutto and Linscott [2021]), an open-source state-of-the-art neural network chess engine trained via self-play reinforcement learning and derived from the AlphaZero framework. The study utilized a locally deployed variant of the Lc0 network t3-512x15x16h-distill-swa-2767500. The study employed the python-chess library for dataset generation. The study utilized Stockfish 17.1 at search depth 30 [Romstad et al., 2008], which provides a robust trade-off between computational cost and evaluation reliability. The study employed Leela Chess Zero (Lc0, Pascutto and Linscott [2021]), an open-source state-of-the-art neural network chess engine trained via self-play reinforcement learning and derived from the AlphaZero framework.

Improvements for AI systems

Refine model fine-tuning strategies to explicitly target generalization gaps by incorporating adversarial training on Out-of-Distribution chess positions (XOOD). This will force models to develop first-principle reasoning rather than relying solely on surface-level heuristics that become brittle once the structural patterns of standard play are removed.
Implement a dynamic reasoning augmentation mechanism that adjusts chain-of-thought prompting based on the estimated density of relevant priors for a given position. This addresses the finding that the relative marginal benefit of reasoning diminishes with decrease in availability of relevant priors, ensuring tokens are used as engines of first-principles derivation when necessary, not just retrieval optimizers.
Develop input encoding methods that mitigate the observed tokenization bottleneck by using direct spatial encodings instead of relying solely on Byte-Pair Encoding for board states. This aims to prevent the model from suffering a catastrophic breakdown when encountering atypical FEN strings, thereby allowing it to maintain robust fluid intelligence outside of crystallized patterns.
Create a tiered deployment system where models are automatically routed based on input complexity estimates derived from prior analysis of position familiarity (WD, ND, OOD). This ensures that high-stakes decisions are only made by models operating in the WD condition, while lower-risk tasks can utilize reasoning augmentation when operating in ND or OOD settings.

Abstract

Large Language Models (LLMs) exhibit remarkable capabilities, yet it remains unclear to what extent these reflect sophisticated recall or genuine reasoning ability. We introduce chess as a controlled testbed aimed at disentangling these faculties. Leveraging the game's structure and scalable engine evaluations, we construct a taxonomy of positions varying in density of relevant priors -- ranging from common states solvable by memorization to completely novel ones requiring generalization. Crucially, our approach achieves this distinction without requiring explicit knowledge of the models' training data. Applying this taxonomy, we combine a longitudinal analysis of the GPT lineage with a rigorous evaluation of contemporary models, including Claude Opus and Gemini. Our analysis reveals a steep gradient: performance consistently degrades as the density of relevant priors decreases. Notably, for tasks with few relevant priors, base model performance regresses to the random-play baseline, and all models propose illegal moves at sharply increasing rates, indicating that they fail to reliably apply the fixed rules of the game in unfamiliar positions. While newer models improve, progress slows significantly for tasks with sparse priors. Furthermore, while reasoning-augmented inference improves performance, its relative marginal benefit per token decreases in the absence of relevant priors. These results suggest limitations in systematic generalization, highlighting the need for mechanisms beyond scale to achieve robust performance when deprived of relevant priors.