Neither Stochastic Parroting nor AGI: LLMs Solve Tasks through Context-Directed Extrapolation from Training Data Priors

arXiv:2505.23323 · cs.CL · Submitted 2025-05-29 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Neither Stochastic Parroting nor AGI".

Jane: Here's the gist of the paper. The gist: The middleground view is that LLMs extrapolate from priors from their training data while using context to guide the model to the appropriate priors, which they call ‘context-directed extrapolation’.

Jane: Context-directed extrapolation is achieved through examples in base models, leading to in-context learning, while instruction tuning allows LLMs to perform similarly based on prompts rather than explicit examplesThe idea that LLMs are just “stochastic parrots” is challenged by their ability to learn in-contextFor example, research by Bonial et al.

10: shows that LLMs can be taught entirely novel phrases, or Multi-Word Expressions (MWEs), on the flySince these novel MWEs could not have been in the training data, this performance demonstrates a capability beyond simply predicting the next most likely wordThe responses LLMs generate result from the combination of priors extrapolated from all aspects of their trainingTherefore, LLMs, like all machine learning systems, do generate output by relying on priors extracted from their training data, whether from pre-training, instruction tuning, or safety alignment Defining Reasoning and Limitations 1.

Title and authors: Tom: The title itself really sets up this tension between seeing them as parrots and seeing them as something more, and this paper wants to settle into that middle ground. It's not about proving they are AGI or just parrots, it's about how they actually work today.

Jane: They’re calling their mechanism "context-directed extrapolation," which is the central concept here. Think of it like a librarian who knows everything in a massive library but needs you to point them to the specific section you want to read right now.

Lu: That context direction is what makes in-context learning possible, where providing examples changes how the model uses its pretraining data priors for that specific task. They're showing how instruction tuning does something similar, targeting relevant priors without needing explicit examples in every prompt.

Meng: It’s interesting from a deployment standpoint because it suggests we can guide their behavior by crafting the input context rather than just changing the underlying training process itself to get better reasoning.

Lalam: And they’re pointing out that even with this context, there are still limits to what the model can actually do, specifically on tasks that require true mastery of a new set of concepts.

The paper's summary: Tom: Okay, so the summary is saying they acknowledge the potential for advanced general-purpose reasoning but argue that current performance isn't there yet. They are focusing on what LLMs can actually do in practice versus what they *could* theoretically do.

Jane: They use examples of models failing on tasks humans solve easily to show where the limits lie, and they link this failure directly back to their context-directed extrapolation model. It explains why they can’t just magically become human-level reasoners overnight.

Lu: Specifically, they look at reasoning through Krathwohl's taxonomy, focusing on "Remember," "Understand," and "Apply." They suggest LLMs are mostly limited to the Understand level, which involves determining the meaning of instructions and things like inferring or comparing concepts.

Meng: So it’s less about complex planning or novel problem-solving outside their learned patterns, and more about accurately interpreting the context we give them. That makes sense when you think about what we actually need for reliable tasks at scale.

Lalam: The paper emphasizes that this reliance on pretraining data means that when the prompt provides enough contextual direction—whether through examples or instruction tuning—the model can effectively extrapolate from those priors, which is a form of understanding as defined in their framework.

The paper's improvements: Tom: The authors suggest a few ways to frame this view better. They are pushing for us to adopt this context-directed extrapolation framework as the most realistic way to understand current LLM capabilities, rather than chasing the more extreme views.

Jane: They want us to stop worrying so much about whether these models are deceptive or have secret agency, and instead focus on how their context-guided extrapolation can be made controllable and predictable for better post-training work.

Lu: One improvement they point to is using instruction tuning specifically to target relevant priors without needing examples, which they say is a key way these models leverage contextual information for task solving. They also suggest counterfactual testing to see if performance is due to generalizable skills or just specific default behaviors.

Meng: I'm interested in that counterfactual testing idea because I've seen how quickly models can get stuck in those default patterns when the context is slightly off, and seeing where they break would be useful for setting deployment thresholds.

Lalam: They also suggest a way to mitigate hallucinations: when the prompt doesn't give enough context, instead of guessing wildly, the model should default to broader pretraining data priors rather than making up things that aren't there.

Conclusion: Tom: So wrapping up, the main point of "Neither Stochastic Parroting nor AGI: LLMs Solve Tasks through Context-Directed Extrapolation from Training Data Priors" is that this framework gives us a solid way to view where LLMs stand right now. They are solving tasks by using context to guide their existing knowledge base.

Jane: It shifts the focus away from existential threats or sudden human-level reasoning and toward understanding exactly how these models use their training data priors under specific instructions. It's about controllability and predictability for future development.

Lu: I think the biggest contribution is framing reasoning as being mostly limited to the "Understand" level, which sets a realistic expectation for what we can ask them to do right now versus what we might hope for later.

Meng: From my side, it’s good that this framework helps us predict where things will break when we push them into new operational environments, because if you know the prior they're pulling from, you know the risk profile.

Lalam: I agree, and I'm confident in this direction. The ability to solve tasks based on context-directed extrapolation is what defines their current success, and it’s a mechanism we can actually work with.

Tom: So to summarize the paper "Neither Stochastic Parroting nor AGI: LLMs Solve Tasks through Context-Directed Extrapolation from Training Data Priors," they argue that LLMs solve tasks using context-directed extrapolation from training data priors, which is a realistic middle ground for understanding their capabilities.

Jane: It means we should stop looking for sudden, unexplainable intelligence and start focusing on how we can better engineer the context—whether through examples or tuning—to get reliable results.

Lu: The implication is that future research into advanced reasoning should be guided by this understanding of what LLMs can actually do right now, setting expectations appropriately for the technology's current maturity.

Meng: And for deployment, it means we need to build in safeguards around context provision because if the context isn't right, they fall back on those default priors that might not be safe or accurate for our specific needs.

Lalam: It’s a framework grounded in the actual mechanics of how these systems operate today, and that's what matters most for practical application moving forward.

Harish Tayyar Madabushi, Melissa Torgbi, Claire Bonial

University of Bath · DEVCOM Army Research Laboratory

cs.CL

Submitted: 2025-05-29

Updated: 2026-10-21

Comments: 14 pages, 1 figure (including appendix)

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: The gist: The middleground view is that LLMs extrapolate from priors from their training data while using context to guide the model to the appropriate priors, which they call ‘context-directed

Key concepts

Context-Directed Extrapolation: This is the core mechanism where LLMs use their pre-trained knowledge as a foundation (priors) but adjust which parts of that knowledge to use based on the specific context given in the prompt. It's not just repeating facts; it's intelligently combining old knowledge with new instructions to solve a specific problem.
In-Context Learning (ICL): This is how models learn new things temporarily without being retrained. In base models, examples provided in the prompt guide the model to perform a task immediately. Instruction tuning allows this to happen through prompts instead of needing explicit examples for every new scenario.
Reasoning Limitations: The paper argues LLM reasoning is limited to 'Understand' (interpreting and classifying meaning) rather than higher-order 'Apply' skills. While they can extrapolate from training data, they struggle with tasks requiring genuine novel planning or mastery outside the provided context, as shown by failures on complex social judgment tests.

Terminology

In-Context Learning — 1× in this paper · explained in 7 episodes
Instruction Tuning — 1× in this paper · explained in 3 episodes

Summary

The gist: The middleground view is that LLMs extrapolate from priors from their training data while using context to guide the model to the appropriate priors, which they call ‘context-directed extrapolation’.

How it works

Context-directed extrapolation is achieved through examples in base models, leading to in-context learning, while instruction tuning allows LLMs to perform similarly based on prompts rather than explicit examplesThe idea that LLMs are just “stochastic parrots” is challenged by their ability to learn in-contextFor example, research by Bonial et al. [10] shows that LLMs can be taught entirely novel phrases, or Multi-Word Expressions (MWEs), on the flySince these novel MWEs could not have been in the training data, this performance demonstrates a capability beyond simply predicting the next most likely wordThe responses LLMs generate result from the combination of priors extrapolated from all aspects of their trainingTherefore, LLMs, like all machine learning systems, do generate output by relying on priors extracted from their training data, whether from pre-training, instruction tuning, or safety alignment

Defining Reasoning and Limitations

Reasoning is focused on the Cognitive Process dimension of Krathwohl’s taxonomy, specifically Remember, Understand, and ApplyLLM reasoning is limited to Understand, which would preclude higher-order processesUnderstanding is defined as “Determining the meaning of instructional messages, including oral, written, and graphic communication,” and it is split into finer grained categories of Interpreting, Exemplifying, Classifying, Summarizing, Inferring, Comparing, and ExplainingThe paper argues that LLM reliance on pretraining data enables-Remembering of Factual KnowledgeExtrapolation from pretraining data is a combination of Remembering, Understanding and hallucinationWhen the pretraining data closely resembles the target task, memory is sufficient for what is essentially stochastic parrotingWhen the prompt includes contextual direction, either through ICL examples in base models, or through the prompt itself in instruction tuned models, then models are able to effectively extrapolate from their pretraining data, demonstrating some Understanding as defined above

Evidence Against Advanced Reasoning

LLMs fail on a range of tasks that humans are able to trivially solveShapira et al. [51] test a variety of models on an established clinical psychology test administered to children that asks them to recognize and pinpoint a faux pas made in a social situation, for example stating to a party host that one does not enjoy apple pie without realizing that this is precisely what the host has prepared for dessertThe best-scoring models on this task (Flan-T5-xl and-xxl) obtain a score of 0.4 on the task, whereas children ages 9-11 score, on average, 0.82 (i.e. children can pinpoint the faux pas in twice as many short anecdotes as models)The Mystery condition reveals that successful performance does not reflect any actual mastery of the planning concepts that can be applied to a new set of items, even though the underlying structure of the problem states and actions hasn’t changed at allThis research (along with later investigations [59, 58]) demonstrates that while the traditional Blocksworld problems are likely well-described on the web and therefore in LLM pretraining data, the Mystery condition reveals that successful performance does not reflect any actual mastery of the planning concepts that can be applied to a new set of itemsThe drop in performance in these cases can be easily explained through our perspective, which posits that LLMs rely on context-guided extrapolation from pre-training priors

Distinguishing Context-Directed Extrapolation

The functional divergence between these two views becomes apparent with instructiontuned modelsOur framework suggests that instruction-tuning allows a model to interpret an instruction as a directive to construct an implicit task context, rather than as a literal sequence to be continuedThis context activates relevant priors from the model’s pre-training data, enabling it to solve the specified taskThe critical evidence supporting this distinction is that instruction-tuned models can solve logical deduction problems without any examplesThis phenomenon is directly accounted for by context-directed extrapolation but cannot be explained by a strict stochastic parroting modelThe demonstrated ability of LLMs to extend prompt-based definitions, apply abstract patterns, and generate context-appropriate interpretations indicates that extrapolation provides access to information not reducible to surface statistics

Conclusion and Recommendations

Near-term research be guided by the more realistic view that current LLM capabilities should be understood as context-directed extrapolationThis perspective will help alleviate fears that LLMs are ‘deceptive,’ as studies into the deceptive capabilities of LLMs simulate intent through prompts under the assumption that such agency will eventually become apparent in the futureUnderstanding the role and importance of context-directed extrapolation in LLMs has the potential to facilitate more effective post-training methodologies that will advance capabilities in a manner that is fully controllable and predictable, increasing trust in LLMsThis framework, which posits that LLMs respond through context-driven extrapolation from training data priors, adequately explains both the capabilities and limitations of LLMsThe research outlined here demonstrates that such preparations should be grounded in understanding the clearest current source of LLM capabilities: context-directed extrapolationThe paper provides a framework for understanding how LLMs function, grounded in existing literature

--- Page 1 ---

Neither Stochastic Parroting nor AGI: LLMs Solve Tasks through Context-Directed Extrapolation from Training Data PriorsHarish Tayyar Madabushi University of Bath Bath, United Kingdom htm43@bath.ac.uk Melissa Torgbi University of Bath Bath, United Kingdom mat66@bath.ac.uk Claire Bonial DEVCOM Army Research Laboratory United States of America claire.n.bonial.civ@army.

Improvements for AI systems

Context-Directed Extrapolation Framework Implementation: The AI system will be engineered to explicitly utilize examples in base models to guide extrapolation, which is described as context-directed extrapolation from training data priors. This allows the model to solve tasks by using priors extrapolated from pre-training data from the several theories put forward on how ICL functions.
Instruction Tuning for Prior Targeting: The system will use instruction tuning not just for following directives, but specifically to enable models to target relevant priors without the need for such examples, as suggested in the paper. This mechanism allows instruction-tuned LLMs to perform similar tasks based on prompts rather than explicit examples, which is described as how they leverage contextual information for task-solving.
Distinguishing Reasoning Levels: The system will be evaluated against a cognitive taxonomy where it is limited to Understand and not Apply, as argued in the paper. This means the AI's capability will be assessed on its ability to explain why items are members of a class, rather than performing novel generalization outside the provided context.
Counterfactual Testing for Generalization Assessment: To test true inherent abilities, the system will be subjected to counterfactual tests, as recommended in the conclusions. This helps determine if performance is due to non-transferable, default-condition-specific behaviors or genuine generalizable reasoning skills.
Hallucination Mitigation via Contextual Grounding: When outputs deviate from reality, the system will be designed such that when the prompt lacks necessary context, outputs default to broader pre-training data priors rather than hallucinating. This addresses hallucinations by ensuring that hallucinations are produced only when the extrapolation from pre-training data leads to an answer outside of the correct range.

Abstract

This position paper argues that many behaviours framed as either "stochastic parroting" or "advanced reasoning" are better explained as context-directed extrapolation from training-data priors, and that ML and NLP research communities would benefit from evaluations that distinguish remembering and basic extrapolation from genuine application-level generalisation. Our central contribution is an actionable evaluation checklist and design patterns that make these distinctions testable in practice. Our framework posits that LLMs extrapolate from training data priors while using context to guide the model to the appropriate priors; we call this "context-directed extrapolation." Specifically, this context direction is achieved through examples in base models, leading to in-context learning, while instruction tuning allows analogous task specification via prompts rather than explicit examples. We argue that adopting this more realistic framework and controls for improving evaluation would better calibrate claims about LLM "reasoning," refocus research on how LLM performance interacts with training data and generalisability to out-of-domain tasks, and reduce miscalibration in deployment and policy decisions.