Encoding Context into LLM Weights: What Makes It Context?

cs.CL, cs.LG

Submitted: 2026-10-13

Updated: 2026-10-13

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Sources

• GPT-4 Technical Report
• ATLAS: Learning to Optimally Memorize the Context at Test Time
• Training Plug-n-Play Knowledge Modules with Deep Context Distillation
• Doc-to-LoRA: Learning to Instantly Internalize Contexts
• Context Length Alone Hurts LLM Performance Despite Perfect Retrieval
• Cartridges: Lightweight and general-purpose long context representations via self-study
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• The Llama 3 Herd of Models
• Measuring Massive Multitask Language Understanding
• RULER: What's the Real Context Size of Your Long-Context Language Models?
• Detecting Unintended Memorization in Language-Model-Fused ASR
• Adam: A Method for Stochastic Optimization
• GradMem: Learning to Write Context into Memory with Test-Time Gradient Descent
• Test-Time Training with KV Binding Is Secretly Linear Attention
• Locas: Your Models are Principled Initializers of Locally-Supported Parametric Memories
• Fast Model Editing at Scale
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• Beyond Perplexity: A Behavioral Evaluation Framework for Deployment-Memory Claims in LLM Test-Time Training
• End-to-End Test-Time Training for Long Context
• Modular TTT: Rethinking Test-Time Training as Composable Modules

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering