What It Costs to Compose, Rebuild, and Correct Precomputed Memory
cs.CL, cs.LG
Submitted: 2026-08-31
Updated: 2026-08-31
License: http://creativecommons.org/licenses/by/4.0/
The gist: Language models can answer from precomputed memory, a model's saved reading of a body of material, reused across requests instead of read again at each.
Terminology
Abstract
Language models can answer from precomputed memory, a model's saved reading of a body of material, reused across requests instead of read again at each. This paper maps where that practice preserves correctness and the conditions under which it fails. Across experiments on Llama-3.1-8B-Instruct using both saved key-value caches and trained compressions of them, precomputed memory degrades when assembled from separately prepared parts, stays current only through rebuilds costing a large fraction of full preparation in our measurements, and ignores corrections served beside it conditional on phrasing. If precomputed memories can be served alongside one another, be cost-efficiently rebuilt, and be superseded by new information arriving in real-time, they can serve as a way to avoid re-feeding context to a model over repeated queries. The implication of our results for a deployed system that deals with a variety of queries is that precomputed memories are best rebuilt on the cadence at which new information changes what the memory was originally computed from. Both warm-rebuilding trained compressions of key-value caches and serving specifically-phrased updates beside a memory, as pasted text or injected cache state, show particular promise for keeping precomputed memories current, the latter as an interim measure between rebuilds, and we measure the cost and name the remaining questions associated with each.
Sources
- LongHealth: A Question Answering Benchmark with Long Clinical Documents
- Cartridges: Lightweight and general-purpose long context representations via self-study
- The Llama 3 Herd of Models
- Cartridges at Scale: Training Modular KV Caches over Large Document Collections
- LoRA: Low-Rank Adaptation of Large Language Models
- InferScale: GPU-Native KV Injection for Personalized LLM Serving
- Block-Attention for Efficient Prefilling
- Evaluating Very Long-Term Conversational Memory of LLM Agents
- When Stale Constraints Go Unchecked: Budgeted Verification Failures in Inherited Agent Memory
- Gemma 2: Improving Open Language Models at a Practical Size
- Step Out and Seek Around: On Warm-Start Training with Incremental Data
- FreshDiskANN: A Fast and Accurate Graph-Based ANN Index for Streaming Similarity Search
- Towards Reliable, Generalizable, and Specific In-Context Knowledge Editing via Multi-Objective Reinforcement Learning
- Efficient Streaming Language Models with Attention Sinks
- Qwen3 Technical Report
- APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encoding
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering