Towards Reliable, Generalizable, and Specific In-Context Knowledge Editing via Multi-Objective Reinforcement Learning
cs.AI, cs.LG
Submitted: 2026-08-25
Updated: 2026-10-02
Comments: Our work proposes a multi-objective reinforcement learning algorithm that optimizes prompt construction for reliable, generalizable, and specific in-context knowledge-editing
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large Language Models (LLMs) are powerful but limited by static parametric knowledge that becomes outdated once pretraining ends.
Terminology
Abstract
Large Language Models (LLMs) are powerful but limited by static parametric knowledge that becomes outdated once pretraining ends. Knowledge editing addresses this problem by updating model behavior on target facts without full retraining. In particular, in-context knowledge editing has gained attention because it is training-free and readily applicable to black-box LLMs. Recent reinforcement learning (RL)-based approaches improve over fixed retrieval strategies by adapting prompt construction to the quantity-quality trade-off. Despite initial success, they fail to model the prompt as a structured entity under the distinct and often competing objectives of reliability, generality, and specificity. Previous methods largely optimize a single objective and make decisions over only part of the prompt construction process, thereby overlooking both the balance of different objectives and the global organization of demonstrations. We propose Multi-Objective In-context Knowledge Editing (MO-IKE), a multi-objective RL algorithm that formulates prompt construction for in-context knowledge editing as a Constrained Markov Decision Process. MO-IKE trains a dynamic retriever to optimize competing objectives in knowledge editing, enabling more balanced and globally coherent prompt construction. On Llama-3.2, MO-IKE improves edit success (reliability) from 85.0% to 92.0%, paraphrase consistency (generality) from 77% to 79%, while increasing retention rate (specificity) by 23.0% compared to prior RL-based methods.
Sources
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
- UniEdit: A Unified Knowledge Editing Benchmark for Large Language Models
- Gold-medalist Performance in Solving Olympiad Geometry with AlphaGeometry2
- Aligned Multi Objective Optimization
- Provably Efficient Sample Complexity for Robust CMDP
- The Llama 3 Herd of Models
- Mistral 7B
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Trust Region Policy Optimization
- Proximal Policy Optimization Algorithms
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Speculative Sampling with Reinforcement Learning
- Knowledge Editing for Large Language Models: A Survey
- Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models
- Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning
- Qwen2 Technical Report
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems