LadderEdit: Edit-Level Residual Compression for Memory-Efficient Lifelong Editing of LLMs
cs.CL, cs.CV, cs.LG, cs.MM
Submitted: 2026-10-08
Updated: 2026-10-08
Comments: EMNLP 2026 Main Conference Long Paper
Code: https://github.com/VisualReasoner/LadderEdit
License: http://creativecommons.org/licenses/by/4.0/
The gist: Lifelong editing of LLMs requires storing thousands of edits after acquisition.
Terminology
Abstract
Lifelong editing of LLMs requires storing thousands of edits after acquisition. A widely used family of approaches attaches one LoRA adapter per edit, which preserves behavior but grows linearly in storage. To address this challenge, we propose LadderEdit, a method that compresses each LoRA adapter after it is acquired. Each edit is first stored at low rank as a cheap sketch. We then check whether this sketch still satisfies the rewrite, generalization, and locality contract on probe prompts. Edits that pass keep the sketch; those that fail are promoted to a higher rank along a ladder until the contract is met. Because every edit retains some representation, coverage is maintained, and only hard edits consume more rank. Across ZsRE, CounterFact, and WikiBigEdit benchmarks on LLaMA-3-8B, Mistral-7B, and Qwen2.5-7B, LadderEdit tracks exact LoRA storage at 5.2x less memory and remains effective at 50,000 sequential edits.
Sources
- Qwen3-VL Technical Report
- The Llama 3 Herd of Models
- Unsupervised Dense Information Retrieval with Contrastive Learning
- Mistral 7B
- ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA
- DoRA: Weight-Decomposed Low-Rank Adaptation
- S-LoRA: Serving Thousands of Concurrent LoRA Adapters
- WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs
- Text Embeddings by Weakly-Supervised Contrastive Pre-training
- WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models
- EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models
- LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Models
- MEMoE: Enhancing Model Editing with Mixture of Experts Adaptors
- Knowledge Editing for Large Language Models: A Survey
- Continual Learning for Large Language Models: A Survey
- Qwen3 Technical Report
- Qwen2.5 Technical Report
- Editing Large Language Models: Problems, Methods, and Opportunities
- MELO: Enhancing Model Editing with Neuron-Indexed Dynamic LoRA
- A Comprehensive Study of Knowledge Editing for Large Language Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering