Local Conflict, Targeted Repair: A Perturbation Analysis of Multi-Domain RL

arXiv:2606.02398 · cs.LG, cs.CL · Submitted 2026-06-01 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Local Conflict, Targeted Repair".

Jane: Here's the gist of the paper. The gist Reinforcement learning post-training improves large language models (LLMs) on individual domains such as mathematical reasoning, code generation, question answering, and creative writing (CW), but training on one domain often degrades performance on others.

Tom: How it works The paper introduces a local perturbation model of multi-domain RL to explain cross-domain interference when full-model gradients are nearly orthogonal.

Jane: This model posits that later-domain training harms an earlier domain mainly through a second-order damage term concentrated in a low-dimensional shared conflict subspace.

Tom: The structural evidence shows that cross-domain interference arises not from global gradient conflict or direct overlap among edited neurons, but by sparse RL edits interacting along shared active computation routes.

Paper summary: Tom: So we've gone through what this paper says about how multi-domain RL post-training can degrade performance and how they propose fixing it with local perturbation theory. What does the title itself suggest?

Jane: The title, "Local Conflict, Targeted Repair," really tells you the story of what they found. It points to a specific mechanism: there is conflict happening locally on certain computational routes, and that conflict can be repaired by targeted intervention >

Lu: It’s about moving past the old explanations based on catastrophic forgetting or global gradient conflict because those explanations don't capture this localized behavior >

Meng: For us deploying models, the implication is that we might not need to constantly retrain a whole model just because it got slightly worse on one specific task >

Lalam: The paper demonstrates that even when global gradients look nearly orthogonal, there's still substantial active-neuron overlap along shared routes that cause trouble >

Tom: So, in simple terms, what does this mean for the people who only listen to the show? What’s the real-world impact?

Jane: It means we can stop thinking about fixing every single problem across all domains at once. Instead, we can identify which specific local routes are causing trouble and apply a short refresh or rollback just to fix that one part >

Lu: It points toward a direction where controlling localized route-level interference is promising for building more stable and scalable multi-domain reinforcement learning systems >

Conclusion: Tom: So we’re wrapping up on this paper, "Local Conflict, Targeted Repair," by those authors—Bau, Belinkov, Sajjad, Durrani—and what they found is that when you train a model for one thing but it messes up another thing later on, the problem isn't a giant global conflict.

Jane: Right. They’re saying the damage happens locally because there are specific computation routes where the updates from different tasks overlap in a very narrow way >

Lu: Exactly. They developed this local perturbation model to show that full-model gradients can look nearly orthogonal on paper, but when you look at the actual paths neurons take, they aren't independent >

Meng: It means we don't have to worry about every single parameter being ruined across all domains just because we trained it once on a new task >

Lalam: I see that the damage is concentrated in a low-dimensional shared conflict subspace, which is essentially where the different tasks fight over the same computational steps >

Tom: So they show how you can fix this by applying a short refresh to just one of those domains, and it geometrically contracts that harmful component >

Jane: That’s what’s so interesting for us. It moves us away from thinking about massive retraining efforts and toward something much more targeted for recovery >

Lu: They also show this works empirically. after training the model in a sequence—code, math, question answering, creative writing—a short refresh on the math part alone raised its score significantly >

Tom: So even though it’s cross-domain interference and they’re dealing with complex gradients, the fix is just a quick re-run on a specific piece of training data >

Jane: It really suggests that for scaling these large models, we might be able to manage multi-task learning more stably by controlling these local interactions >

Lu: And it opens up ideas for alternating refreshes, which they suggest can act like a way to approach a compromise between the different domains instead of forcing one to completely dominate the other >

Meng: If this holds up at scale, it could drastically lower our maintenance costs because we wouldn't need massive compute just to keep the whole system from breaking down >

Tom: It’s about pinpointing exactly where the trouble is happening in the model’s internal structure rather than treating it like a generalized problem >

Jane: And that level of specificity, showing how a small local intervention can fix a big cross-domain issue, that’s what makes this paper so important for anyone building these systems.

Lei Yang, Siyu Ding, *Deyi Xiong

Tianjin University · Baidu Inc.

cs.LG, cs.CL

Submitted: 2026-06-01

Updated: 2026-10-17

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 86/100

The gist: The gist Reinforcement learning post-training improves large language models (LLMs) on individual domains such as mathematical reasoning, code generation, question answering, and creative writing

Key concepts

Local Perturbation Model: This model explains how training on one domain harms another by focusing only on local changes in the neural network. It posits that damage occurs through a second-order term concentrated in a low-dimensional shared conflict subspace, rather than broad, global gradient disagreements across the entire model.
Shared Active Conflict Subspace (SA,B): This is a specific low-dimensional area where updates from two different domains (A and B) interact negatively. The structural analysis shows that cross-domain interference arises because later domain updates concentrate their harmful curvature within this shared subspace, which is linked to the computation routes both domains use.
Short Refresh: A 'refresh' is a targeted, short update applied specifically to the earlier domain (Domain A) after training on a later domain (Domain B). The model proves that this local intervention geometrically contracts the harmful component in the shared conflict subspace, allowing for fast recovery of performance in Domain A without significantly damaging other domains.
Curvature Probes: These are analytical tools used to measure how sensitive the objective function is along different directions. The paper uses them to show that checkpoints from one domain exhibit much higher sensitivity along the direction of conflict than they do along random directions, pinpointing the exact subspace where targeted repair should be applied.

Terminology

Catastrophic Forgetting — 1× in this paper · explained in 9 episodes
Large Language Models (LLMs) — 1× in this paper · explained in 8 episodes
Reinforcement Learning — 1× in this paper · explained in 4 episodes
Reinforcement Learning (RL) — 1× in this paper · explained in 25 episodes

Summary

The gist Reinforcement learning post-training improves large language models (LLMs) on individual domains such as mathematical reasoning, code generation, question answering, and creative writing (CW), but training on one domain often degrades performance on others.

How it works

The paper introduces a local perturbation model of multi-domain RL to explain cross-domain interference when full-model gradients are nearly orthogonal. This model posits that later-domain training harms an earlier domain mainly through a second-order damage term concentrated in a low-dimensional shared conflict subspace. The structural evidence shows that cross-domain interference arises not from global gradient conflict or direct overlap among edited neurons, but by sparse RL edits interacting along shared active computation routes.

Structural Evidence for Localized Cross-Domain Interference

The analysis moves beyond full-model objective conflict to the local active routes through which domain updates are expressed. The theory is supported by three structural conditions: (i) the selected domain-A checkpoint is approximately stationary for LA; (ii) later-domain updates are local and effectively sparse; and (iii) the curvature-sensitive part of the later update is concentrated in a low-dimensional shared active conflict subspace SA,B. Proposition 1 formalizes that later-domain damage is governed by the local curvature of the earlier-domain objective.

Short Refresh Geometrically Contracts the Conflict Component

The paper proves that a short refresh on domain A geometrically contracts the harmful component in this shared conflict subspace. Theorem 1 demonstrates that starting from a degraded checkpoint θ0 = θ∗A + δB, a short refresh on domain A satisfies∥PS(θt − θ∗A)∥2 ≤ (1 − αµA)t∥PSδB∥2. This provides a local explanation for fast recovery: early refresh steps can rapidly remove the harmful component.

Task-Level Validation and Direct Intervention

The theory is validated through two complementary empirical predictions: short-refresh recovery at the task level and a training-free targeted rollback on a coordinate proxy for the conflict subspace. The direct rollback experiment shows that reverting only 2% of MLP neurons selected by the composite score A × M × C raises Math Avg from 59.90 to 61.25, recovering 20.4% of the QA-induced Math loss.

Stability of Other Domains under Near-Orthogonal Gradients

The analysis shows that refresh steps are small and the involved gradient norms are not too large, leading to limited one-step damage to other domains when global gradients are nearly orthogonal. This separation between local conflict and global near-orthogonality explains how Re-Math can recover Math while largely preserving Code, QA, and CW >.

Extension: Alternating Refresh as Local Multi-Objective Optimization

Alternating refresh across domains can be viewed as a local mechanism for approaching a Pareto-stationary compromise among domains under the weighted-sum scalarization view. This suggests that repeated refresh can keep improving for several rounds without implying that all single-domain optima are simultaneously reachable >. The paper concludes that controlling localized route-level interference is a promising direction toward more stable and scalable multi-domain RL >.

Limitations

The work suggests future directions include automating the detection of degraded domains, the selection of refresh order and budget, and examining whether repeated targeted refresh consistently approaches a local Pareto-stationary compromise >. Furthermore, it suggests that identifying the conflict subspace more precisely could lead to a smaller and less redundant intervention budget for future work >. The analysis also extends to other post-training paradigms like on-policy distillation >.

References

[1] Art of Problem Solving. Aime problems and solutions.

[2] A. Bau, Y. Belinkov, H. Sajjad, N. Durrani, F. Dalvi, and J. R. Glass.

[3] H. Chen, N. Razin, K. Narasimhan, and D. Chen.

[4] Z. Chen, V. Badrinarayanan, C. Lee, and A. Rabinovich.

[5] P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei.

[6] T. Chu, Y. Zhai, J. Yang, S. Tong, S. Xie, D. Schuurmans, Q. V. Le, S Levine, and Y Ma.

[7] D. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F Wei.

[8] DeepSeek-AI.

[9] J. Dekoninck, N. Jovanovic, T. Gehrunger, K. Rögnvalddson, I Petrov, C Sun, and M Vechev.

[10] C. He, R. Luo, Y Bai, S Hu, Z L Thai, J Shen, J Hu, X Han, Y Huang, Y Zhang, J Liu.

[11] M. Huan, Y Li, T Zheng, X Xu, S Kim, M Du, R Poovendran, G Neubig, and X Yue.

Improvements for AI systems

Localized Interference Mitigation: Implement a short domain refresh mechanism to selectively recover performance on a degraded domain while bounding collateral damage to others, as shown by Re-Math recovering Math from 57.66 to 66.04 while yielding the best average score of 66.39.
Targeted Damage Rollback: Develop a training-free rollback procedure using a sparse coordinate proxy for the shared active conflict subspace, which can recover a substantial portion of task loss, such as recovering 20.4% of the QA-induced Math loss in Table 6 without full retraining.
Directional Sensitivity Analysis: Utilize directional alignment on shared edited neurons to determine whether later training helps or harms earlier domain performance, allowing the system to understand that the direction of updates on shared routes then determines whether the effect is synergistic or conflicting.
Alternating Refresh for Pareto Compromise: Employ alternating refresh cycles across domains to approximate gradient descent on a weighted local objective, suggesting that repeated refresh can keep improving for several rounds without implying that all single-domain optima are simultaneously reachable.
Attention Unit Intervention: Extend the rollback mechanism to attention units by defining a fine-grained score combining update magnitude and directional conflict, allowing for recovery beyond MLP coordinates when using larger intervention budgets.