The Shape of Wisdom: How Language Models Converge on an Answer

The gist: An answer is better understood as a trajectory than as a single endpoint, and correctness and stability are different properties that can be separated by tracking state, motion, and boundary distance.

Answers are trajectories

Language models do not simply choose an answer at the output layer; in a 9,000-trajectory MMLU study across Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, and Mistral-7B-Instructv0.3, the score of the answer moves across depth in structured ways. We describe each trajectory with three quantities: the current answer margin, the next-layer change in that margin, and the distance from a decision flip. The main empirical picture is that correctness and stability are different; the largest group is unstable-correct, not stable-correct. A traced subset then asks what moves the margin. In stable-correct cases, the average attention scalar points in the correct direction, while the average MLP scalar does not. Span deletion shows that removing answer-supporting text hurts the margin and removing distractor-like text helps it.

State, motion, and boundary distance

After that, we ask what explains the movement. We call these state, motion, and boundary distance. The first result is descriptive but important. State says whether the correct option currently leads. Motion says whether the next layer pushes that margin up or down. Boundary distance says how easily a small movement could change the preferred answer. This coordinate system is deliberately modest. It does not claim a physical phase transition or a discovered basin in hidden-state space. Its value is practical: it tells us whether a final answer was reached by early commitment, late movement, or persistent uncertainty.

Trajectory regimes

We classify each model-prompt row into four operational trajectory types. Stable-correct trajectories end correct and are stable in the tail of the network. Stable-wrong trajectories end wrong and are also stable. Unstable-correct and unstable-wrong trajectories end correct or wrong, but keep enough late movement that they fail the stability rule. The main surprise is how common unstable success is. In the full panel, unstable-correct is the largest group.

Mechanistic accounting of motion

We use a smaller, balanced subset of 600 prompts per model, with 150 prompts from each trajectory type. For those prompts, stored traces record two numbers at each layer: how much the attention blocks move the margin, and how much the MLP blocks move it. The first finding is that these summaries explain a real part of the motion. On held-out prompts, a linear combination of attention and MLP scalars reconstructs layer-to-layer margin drift with R2 = 0.90 for Qwen2.5-7B. This means the scalars are not just decorative diagnostics.

Span interventions and controls

We report the effect as original margin minus margin after deletion. A positive effect means the removed text had been helping the correct answer: once it is deleted, the margin falls. With this sign convention, evidence-labeled spans have mean effect 2.92 logit-margin units. The narrower result is that some marked spans causally move the same margin that defines the trajectory regimes.

Discussion and limits

The paper’s conclusion is not just that four trajectory types exist. The useful conclusion is that final accuracy mixes together at least two different properties: whether the answer is correct, and whether the model has actually settled. In these traces, that means making the correct option pull ahead earlier and keeping it ahead. The takeaway is simple: the best behavior is not just getting the answer right, but getting it right and then staying settled. The useful clues are concrete: attention gives the clearest small layer-by-layer push toward stable-correct answers, MLP replacement gives the largest simulated late boost, and deleting evidence or distractor text moves the same answer advantage in predictable directions. The empirical scope is narrow by design.

--- Page 1 ---

The gist

Language models do not simply choose an answer at the output layer; in a 9,000-trajectory MMLU study across Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, and Mistral-7B-Instructv0.3, the score of the answer moves across depth in structured ways.

Answers are trajectories

Language models do not simply choose an answer at the output layer; in a 9,000-trajectory MMLU study across Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, and Mistral-7B-Instructv0.3, the score of the answer moves across depth in structured ways. We describe each trajectory with three quantities: the current answer margin, the next-layer change in that margin, and the distance from a decision flip.

1. Stable answer selection: Improve model performance by focusing on achieving stable-correct trajectories instead of just final accuracy, as the best behavior is not just getting the answer right, but getting it right and then staying settled. This means training objectives should prioritize minimizing late movement and maximizing stability in the tail of the network.

2. Mechanistic intervention for margin control: Implement prompt engineering that targets specific components to move the margin predictably; for example, removing evidence-labeled spans will lower the correct answer margin, while removing distractor-labeled spans will raise it.

3. Attention alignment for stable correctness: Fine-tune attention mechanisms to exhibit positive average scalars during stable-correct trajectories, as stable-correct trajectories have the clearest positive average attention contribution in the traced panel. This suggests optimizing attention blocks to consistently push the correct option ahead of competitors early in the processing depth.

4. MLP boosting for late margin shifts: Utilize a mechanism that can simulate adding or substituting MLP scalars to a failing trajectory; since MLP substitution has the larger final-margin effect, training should focus on enhancing MLP contributions, especially in later layers, to provide the largest simulated shift in final margin.

When a language model answers a factual question, where does its internal state go? We asked four open models (Llama-3.1-8B, Qwen2.5-7B, Gemma-2-9B and Llama-3.2-1B) 2,256 factual questions whose answers recur, such as the country a person is a citizen of. At every layer, we recorded the state from which the answer is produced. Questions that receive the same answer converge on a shared region of state space, which we summarise by its centre, the answer centroid. The answer becomes readable within a single layer. In every model, the share of questions whose state lies nearest its own answer's centroid jumps at one layer, between 47% and 71% of the way through the network. The centroids follow the answer the model gives rather than the question: the centroid of a country learned from questions about people's citizenship also identifies that country in questions about towns and films. The model also uses the centroids. After the jump, moving the state of a wrongly answered question to the right answer's centroid makes the model give the right answer in 83%-97% of cases late in the network, against 2%-4% for a random move of the same size. Before the jump, moves of the same size almost never work. Near misses, in which the model answers wrongly but picks the right option in multiple choice, can be seen from the onset layer, where the right answer's centroid is already among the nearest to the state. This signal is about as informative as the right answer's rank in the model's final output, and more informative than the model's confidence in its reply.