Kinetic theory for Transformers and the lost-in-the-middle phenomenon

arXiv:2605.09213 · math.AP, cs.LG, math.PR · Submitted 2026-05-09 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Kinetic theory for Transformers and the lost-in-the-middle phenomenon".

Jane: Here's the gist of the paper. The gist The token study causal self-attention dynamics—a toy model for decoder Transformers—which we interpret as a non-exchangeable interacting particle system, and we prove a quantitative mean-field limit result and a next-order characterization of correlations.

Tom: Model Framework The paper views Transformers as "interacting particle systems

18: " and considers a minimal model on the one-dimensional torus T:= R/2πZ, which is "structurally faithful to the ones used in the experiments of

30: " (Page 2).

Jane: The dynamics are described by an evolution equation for token embeddings at layer t: dθj(t) = 1/ZN,j X j−1 k=1 e− λ N (j−k)w′ β (θj(t) − θk(t)) (Equation 1.1), incorporating the positional encoding factor e−λ(j−k)/N known as ALiBi

34: (Page 2).

Tom: This causal structure makes the system "non-exchangeable" (Page 2).

Paper summary: Tom: So we're looking at this paper today, "Kinetic theory for Transformers and the lost-in-the-middle phenomenon." Basically, they take that self-attention mechanism in decoder Transformers and treat it like an interacting particle system.

Jane: That sounds intense. What is the main idea they're trying to get across? Like, what's the big question this paper is answering?

Tom: They are studying causal self-attention dynamics, which they call a toy model for decoder Transformers, and interpreting it as a non-exchangeable interacting particle system. The goal here is to prove a quantitative mean-field limit result and get a next-order characterization of correlations.

Lu: That's the core ambition. They are adapting cumulant expansions to this triangular causal dependency structure of the model, and they're using non-hierarchical methods to estimate correlations with Glauber calculus.

Meng: Wait, so they are trying to get a concrete prediction out of what is inherently a very complex system? That sounds like it could lead to some really useful scaling laws if it works.

Lalam: I see they specifically look at iid uniformly distributed tokens, and for those cases, the limiting correlation equation can be solved in closed form. This gives them a rigorous explanation of the lost-in-the-middle phenomenon that people have empirically seen before.

Jane: The empirical phenomenon is this token retrieval profile being U-shaped as you look at where the source position is in the prompt. So, what does this mean for us who just listen to AI systems?

Tom: It means that for iid uniformly distributed tokens, the token retrieval profile, as a function of the source position in the prompt, is U-shaped, with primacy and recency and a unique interior minimum under an explicit smallness condition.

Lu: The paper proves this by analyzing what they call the leading correction St(sigma0), which they show is U-shaped through solving the Volterra–Hardy equation. This gives them a rigorous explanation of that empirical U-shape you mentioned, Jane.

Meng: If it's really about how fast the system evolves, I wonder what this means for deployment at scale. Does this mean we can predict where information is most likely to be found in a prompt before we even run the inference?

Paper summary: Jane: It suggests that for large language models, the way they retrieve information isn't just random; there's a predictable structure based on where the source token sits. This predictability, when scaled up, is what they are quantifying with their mean-field limit.

Tom: They establish a qualitative mean-field limit where the rescaled index sigma equals j over N plays the role of a continuous reading cursor variable, leading to a limiting directed graphon k lambda of sigma and sigma prime.

Lu: That graphon, k lambda (sigma, sigma prime), is what captures the structure in the continuum description. And they show that this relates to time correlations Cov(e in theta N (t), e - i theta i* (zero)), which vanish at the level of the mean-field description <ref:2605.09213#pg2>.

Jane: So, when we look at that limiting graphon, it seems like for most things, the relevant signal gets washed out by the mean-field approximation. That means they have to go deeper than just the average behavior.

Tom: Exactly. But they do go deeper with next-order characterization of correlations, which is where they characterize how this dependence on source position sigma0 actually happens through a forcing term determined by A phi (Equation one point one eight) <ref:2605.09213#pg2>.

Meng: That forcing term is key because it shows that the fluctuation carried by the source token at position sigma0 gets injected into the dynamics of C phi, weighted by that limiting graphon k lambda of sigma and sigma zero. That sounds like a precise mechanism for how context matters.

Lalam: I can confirm what they prove is Theorem one point three: for iid uniformly distributed prompts, the soft accuracy sigma0 equals AN (t, sigma0) is U-shaped with a unique interior minimum for all N large enough <ref:2605.09213#pg2>.

Jane: That's the big payoff for people who just use these models. It means that when you put information in a prompt, there’s a specific spot in the middle that is least likely to be used by the AI, even if you have plenty of context around it.

Tom: The authors confirm this by showing that the function sigma0 -> St(sigma0) has a unique global minimum in (zero one) <ref:2605.09213#pg2>. This rigorously explains why we see primacy and recency effects.

Lu: And they used Glauber calculus to get these sharp bounds on covariances and third cumulants, which are really crucial for proving those final error bounds in Theorem one point two, like Equation five point four and Equation five point two which bound the error in the soft accuracy by terms involving N minus delta alpha one p e c t n C <ref:2605.09213#pg3>.

Paper summary: Meng: Those error bounds are important because they tell us how much noise we can expect when we try to apply this theory to real traffic, not just toy models with iid tokens. It gives us a measure of uncertainty.

Jane: So, to sum up, the paper explains the lost-in-the-middle phenomenon not as a quirk of training data but as a feature arising from the underlying kinetic theory of interacting particle systems in these Transformer models.

Tom: And it extends those results to arbitrary smooth and even periodic interaction kernels and general causal self-attention dynamics. That means this isn't just for one specific setup; it has broader applicability across different model designs.

Lu: I think the real impact is showing that we can derive these kinds of structural properties from first principles kinetic theory, rather than just observing them in simulation or empirical experiments eighteen <ref:2605.09213#pg2>.

Meng: From my side, it's interesting because they are using a minimal model on the one-dimensional torus T:= R/2pi Z, which is structurally faithful to experiments in thirty <ref:2605.09213#pg2,a minimal model on the one-dimensional torus T := R>. That faithfulness is what gives the mathematical rigor they need.

Jane: It really connects the abstract math—the particle system—to something we see in actual LLM performance, like why retrieval profiles look that way.

Tom: So, if you're thinking about future work, the authors extend these results to more general causal self-attention dynamics and arbitrary interaction kernels. That opens up a lot of avenues for applying this kinetic theory to other complex neural network architectures.

Lu: It certainly suggests that this framework is powerful enough to handle systems where the interactions aren't just simple cosine interactions on a torus, which is definitely an ambitious goal.

Meng: For deployment, I guess the next step would be figuring out how to estimate these correlation terms practically without needing all those high-order cumulants. That would be a huge practical hurdle.

Jane: It sounds like this paper gives us a very solid mathematical foundation for understanding the behavior of large models beyond just running them and hoping for the best.

Conclusion: Tom: So we're wrapping up this deep dive into "Kinetic theory for Transformers and the lost-in-the-middle phenomenon." We looked at how they treat these giant decoder models like interacting particle systems to explain some weird behavior in their outputs.

Jane: Right, so the big picture is that they are taking something super complex—these huge transformer models—and looking at it through a lens of physics, specifically kinetic theory, to figure out why things sometimes behave the way they do when retrieving information from a prompt.

Lu: Exactly. They show you how adapting cumulant expansions to this triangular causal structure allows them to prove a quantitative mean-field limit and give us next-order characterization of those correlations between tokens.

Meng: From my side, what I’m really looking at is that they are using these very rigorous mathematical tools—Glauber calculus for estimating correlations—to explain something that was just an empirical observation in some previous work. It moves it from "it looks like this" to "this is why."

Lalam: And the main result, Theorem one point three, confirms the lost-in-the-middle phenomenon for iid prompts: the retrieval profile has a U shape with a unique interior minimum when you look at it as a function of source position. I'm pretty sure of that answer.

Jane: That U shape is what everyone sees in those old experiments, right? So, what does this actually change for someone who just uses these models every day?

Tom: Well, it means the way an AI pulls context isn't totally random; there's a specific spot in the middle of a prompt that it generally ignores. It’s not just noise; it’s a structure dictated by the underlying physics of how those tokens interact.

Lu: The next thing they proved is that this dependence on source position is actually driven by a forcing term in their correlation equations, which connects the specific token you start with to the overall dynamics of the system.

Jane: So, if we think about this for production models, does it mean we can actually predict where an AI will find useful information before it even starts reading?

Meng: Not directly, not yet. But understanding these correlation terms gives us a way to measure how much uncertainty there is in that retrieval profile, which is something I’d need if I wanted to deploy this reliably at scale.

Tom: The authors did extend this framework to more general causal self-attention dynamics and even arbitrary interaction kernels, so it’s not just stuck on one specific type of attention mechanism.

Jane: That's a big deal for the future, because it means we can apply these kinetic theory ideas to other kinds of neural networks that we haven't even looked at yet.

Lu: It opens up a lot of avenues for applying this kind of kinetic theory to other complex systems beyond just language models. We’re showing the method has broad applicability across different architectures.

MITIA DUERINCKX, BORJAN GESHKOVSKI, STEFANO ROSSI

University Libre de Bruxelles · inria · Sapienza Università di Roma

math.AP, cs.LG, math.PR

Submitted: 2026-05-09

Updated: 2026-10-20

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: The gist The token study causal self-attention dynamics—a toy model for decoder Transformers—which we interpret as a non-exchangeable interacting particle system, and we prove a quantitative

Key concepts

Non-exchangeable Interacting Particle System
This framework views Transformer token embeddings as particles that interact based on their causal dependencies. Unlike simpler models where particles can be swapped freely, this system captures the directed flow of information inherent in how attention mechanisms process sequential data, making the dynamics complex and non-trivial.
Mean-Field Limit
This mathematical technique simplifies a complex system by assuming that each particle interacts with an average field rather than every other individual particle. In this context, it allows researchers to derive a tractable continuous equation describing the overall behavior of token retrieval as the number of tokens (N) becomes very large.
Lost-in-the-Middle Phenomenon
This is an empirical observation where models show weaker performance when retrieving information from tokens located in the middle of a prompt, compared to those at the beginning or end. The paper provides a rigorous kinetic theory explanation, proving this pattern arises naturally from the underlying physics of the particle system dynamics.

Terminology

Summary

The gist The token study causal self-attention dynamics—a toy model for decoder Transformers—which we interpret as a non-exchangeable interacting particle system, and we prove a quantitative mean-field limit result and a next-order characterization of correlations.

Model Framework

The paper views Transformers as interacting particle systems [18] and considers a minimal model on the one-dimensional torus T:= R/2πZ, which is structurally faithful to the ones used in the experiments of [30] (Page 2). The dynamics are described by an evolution equation for token embeddings at layer t: dθj(t) = 1/ZN,j X j−1 k=1 e− λ N (j−k)w′ β (θj(t) − θk(t)) (Equation 1.1), incorporating the positional encoding factor e−λ(j−k)/N known as ALiBi [34] (Page 2). This causal structure makes the system non-exchangeable (Page 2).

Mean-Field Limit and Correlations

The analysis aims to prove that the retrieval observable, defined in Equation 1.5, satisfies the U-shape observed empirically in [30] (Page 3). The first result is Theorem 1.1, which establishes a Qualitative mean-field limit where the rescaled index σ = j/N plays the role of a continuous reading-cursor variable, leading to a limiting directed graphon kλ(σ, σ′) (Equation 1.9) (Page 3). The error estimates in Theorem 1.2 show that the relevant signal is carried by time correlations Cov(e inθN (t), e−iθi∗ (0)), which vanish at the level of the mean-field description (Page 4).

Characterization of Correlations

The spatial dependence of the accuracy function is captured by the cross-correlation C N φ, which is described by Equation 1.14 (Page 4). Theorem 1.2 characterizes this dependence, showing that the cross-correlation evolves according to a linear equation with a forcing term determined by A φ (Equation 1.18) (Page 5). The mechanism responsible for the dependence of the retrieval profile on the source position σ0 is this forcing injects the fluctuation carried by the source token at position σ0 into the dynamics of C φ, and is weighted by the limiting graphon kλ(σ, σ0) (Page 5).

Lost in the Middle Phenomenon

Theorem 1.3 proves that for iid uniformly distributed prompts, "the soft accuracy σ0 -> AN (t, σ0) is U-shaped with a unique interior minimum for all N large enough" (Page 6). This result is derived by analyzing the leading correction St(σ0) defined in Equation 1.21, which is shown to be U-shaped through the explicit solvability of the Volterra–Hardy equation (Equation 1.19) (Page 6).

Error Estimates and Bounds

The error estimates are derived using Glauber calculus adapted to the triangular causal structure of the dynamics, yielding sharp bounds on covariances and third cumulants (Page 10). Specifically, Lemma 3.1 provides bounds for Cov φ (θi(t)), φ θj (t′) (Equation 3.3) and κ1,1,1[φ(θi(t)), φ θj (t′), φ θk(t′′)] (Equation 3.4) (Page 10). These estimates are crucial for proving the final error bounds in Theorem 1.2, such as Equation 5.4 and Equation 5.2, which show that the error in the soft accuracy is bounded by terms involving N−δ∧1p e Ct⟨n⟩ C (Page 17).

Conclusion

The entire analysis concludes with the proof of Theorem 1.3, demonstrating that "the function σ0 -> St(σ0) has a unique global minimum in (0, 1)" (Page 6). This confirms that the empirically observed lost-in-the-middle phenomenon is rigorously explained by the underlying kinetic theory of interacting particle systems. The paper also extends results to arbitrary smooth and even periodic interaction kernels and general causal self-attention dynamics (Page 12).

--- Page 1 ---

KINETIC THEORY FOR TRANSFORMERS AND THE LOST-IN-THE-MIDDLE PHENOMENON

MITIA DUERINCKX, BORJAN GESHKOVSKI, AND STEFANO ROSSI

Abstract. We study causal self-attention dynamics—a toy model for decoder Transformers—which we interpret as a non-exchangeable interacting particle system. Adapting cumulant expansions to the triangular causal dependency structure of the model, and appealing to non-hierarchical methods to estimate correlations using Glauber calculus, we prove a quantitative mean-field limit result and a next-order characterization of correlations. For iid uniformly distributed tokens, the limiting correlation equation can be solved in closed form and we obtain a rigorous explanation of the empirically observed lost-in-the-middle phenomenon: the token retrieval profile, as a function of the source position in the prompt, is U-shaped, with primacy, recency, and a unique interior minimum under an explicit smallness condition.

--- Page 2 ---

2 M. DUERINCKX, B. GESHKOVSKI, AND S. ROSSI

A convenient framework for addressing this question from a mathematical lens is to view Transformers [38]—the neural network architecture underlying large language models—as interacting particle systems [18]. This has already proved fruitful for encoder Transformers, which are exchangeable particle systems, where empirically observed representation collapse phenomena [41, 12, 31, 11] can be rigorously proven and seen as clustering of the particles over time [17, 18, 9, 33, 16, 6]. Results of this nature have since extended to substantially more general settings [7, 29, 15, 28, 1, 19, 8].

--- Page 3 ---

LOST IN THE MIDDLE

By the Poisson summation formula, this admits the Fourier representation AN (t, σ0) = p π/2 M X n∈Z e− π/2 2M squared n squared E h e in(θN (t)−θi∗ (0))i.

--- Page 4 ---

4 M. DUERINCKX, B. GESHKOVSKI, AND S. ROSSI

Figure 1. The experiment in [30] that motivates the paper. The two panels on the left show how the answer-containing document is moved through the prompt and how the amount of surrounding text is changed. The curve on the right is the empirical lost-inthe-middle profile: retrieval is strongest near the beginning and at end of context, and weakest in middle. Reproduced from Figures 3, 4, and 1 of [30], respectively, with permission from the authors.

--- Page 5 ---

LOST IN THE MIDDLE

By the Poisson summation formula, this admits the Fourier representation AN (t, σ0) = p π/2 M X n∈Z e− π/2 2M squared n squared E h e in(θN (t)−θi∗ (0))i.

--- Page 6 ---

6 M. DUERINCKX, B. GESHKOVSKI, AND S. ROSSI

By the Poisson summation formula, this admits the Fourier representation AN (t, σ0) = p π/2 M X n∈Z e− π/2 2M squared n squared E h e in(θN (t)−θi∗ (0))i.

--- Page 7 ---

LOST IN THE MIDDLE

By the Poisson summation formula, this admits the Fourier representation AN (t, σ0) = p π/2 M X n∈Z e− π/2 2M squared n squared E h e in(θN (t)−θi∗ (0))i.

--- Page 8 ---

8 M. DUERINCKX, B. GESHKOVSKI, AND S. ROSSI

Figure 3. Simulation of the particle system (1.1) for N = 64 and β = λ = 1, initialized near four small angular clusters. Crosses mark the initial angles θk(0), while hollow points mark the evolved angles θj (t); the labels follow the first and last positions. The plot is read by comparing evolved angles with initial ones: for a fixed output position j, proximity of θj (t) to θk(0) is the trajectory-level analogue of alignment with source position k, and for j = N this is essentially the distance entering the retrieval observable. Two effects are visible. The drift of the terminal particle θN(t) toward the first particle θ1(t) shows the primacy effect; collapse toward the first token is proved in [28], and is consistent with empirical work on attention sinks [40, 21, 2]. All the while, at short times the terminal particle remains close to the packet of nearby highindex particles initialized near θN(0); this coherent terminal packet somewhat reflects the recency side of the U-shaped profile.

--- Page 9 ---

LOST IN THE MIDDLE

By Theorem 1.

Improvements for AI systems

  1. Primacy effect optimization: The system can be tuned to prioritize relevant information located at the very beginning of a long prompt by leveraging the mathematical result that "Primacy: St(σ0) -> +∞ as σ0 -> 0."

  2. Recency bias mitigation: To counteract the tendency for models to focus on recent tokens, the system architecture can be modified such that "S′t(1−) > 0" is maintained, preventing the model from exclusively favoring late-stage context.

  3. U-shaped accuracy prediction: The AI system can be designed to predict retrieval success by analyzing the soft accuracy function AN (t, σ0), which is proven to be U-shaped with a unique interior minimum under smallness conditions.

  4. Contextual relevance mapping: By utilizing the cross-correlation C N φ, the system can map how the retrieval accuracy depends on specific source positions in the prompt, allowing for better understanding of internal context utilization.

Sources

Related papers