Output Vector Editing for Memorization Mitigation in Large Language Models

arXiv:2606.18767 · cs.CL · Submitted 2026-06-17 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Output Vector Editing for Memorization Mitigation in Large Language Models".

Jane: Here's the gist of the paper. The gist The proposed output vector editing method modifies what neurons contribute to the residual stream rather than whether they are active, achieving up to 87.9% suppression on OLMo-7B by introducing a distractor in vocabulary space while leaving activations unchanged How it works The core of the method is output vector editing, a constrained-optimization weight edit that locates a small set of MLP neurons responsible for a memorized continuation and minimally modifies their output vectors to introduce a distractor in vocabulary space, redirecting their residual-stream contributions while leaving activations unchanged This is achieved by modifying the output vector vi of MLP neurons implicated in a memorized continuation The closed-form solution is a rank-one update along a chosen distractor token’s unembedding direction, adapting the rank-one weight-update mechanism of ROME from matrix-level factual editing to perneuron output redirection Unlike activation steering, our edit is computed once per memorized sequence and applied directly to the weights; at inference the model runs unchanged Memorization Mining and Localization The process begins with mining memorized content directly from each model’s pretraining corpus using teacher-forced prediction, where outputs agree at the first continuation token, which is the target of our editing For OLMo-7B, 6,831 unique sequences were mined from its pretraining corpus Localization is a two-step process that identifies contributing neurons in two stages Step 1 involves layer importance, where the final residual stream decomposes additively across all layers, and components are projected through the unembedding matrix to calculate an LI value indicating boost or dampening of the target token Step 2 scores each neuron by its L1 distance to r post, where a high score is received if the contribution ai·vi is close to r post in L1 distance Only neurons whose unembedded output vector (vi·WU) ranks the target token either in the top 50 of the vocabulary or in the bottom 50 are selected for editing Edit Modes and Performance Trade-offs Four edit modes span a spectrum from aggressive suppression to minimal redirection The modes are defined by the choice of distractor token s, which defines three promotion modes and a fourth mode that dampens the target directly The four edit modes include end-of-sequence (EOS), redaction marker (redact), next-best, and direct suppression Evaluation on OLMo-7B showed a success–locality trade-off across three operating points Conservative editing (redact) at 31.7% suppression has near-zero locality cost, while practical editing (next-best, k=5) at 81.5% has no catastrophic edits Mechanistic Findings and Limitations The research identified a mechanistic boundary at ∼14% of sequences unreachable by MLP-only editing Attention-layer analysis showed these failures are not driven by attention overall, but ablating the top contributing attention heads at the target position recovers 56–64% of EOS k=10 failures and 53–60% of Next-best k=5 failures The paper also noted that universal failures, approximately 921 of 6,831 memorized sequences (∼14%) resist every MLP-based configuration These universal failures are often driven by strong sequential dependencies where memorization is distributed redundantly beyond the top-k budget Furthermore, aggressive modes like EOS (k=10) can incur tail risk, with 47.8% of edits having a median shift in perplexity of +0.88 Cross-Model Generalization Success rates scale with model size rather than family, as SmolLM-360M (34.9%) and OLMo-1B (52.4%) sit well below OLMo-7B's 85.7% success rate Locality costs are architecture-dependent, with the catastrophic-outlier tail on OLMo-7B aggressive modes not appearing on smaller models like OLMo-1B or Llama2-7B The paper concludes that output vector editing is a weight-level intervention that modifies what neurons contribute to the residual stream rather than whether they are active How it works The paper presents output vector editing, a neuron-level method for mitigating memorization in LLMs that modifies what neurons contribute to the residual stream rather than whether they are active It was evaluated across four models from two architecture families and different sizes (SmolLM360M, OLMo-1B, OLMo-7B, Llama2-7B), with a deeper grid on OLMo-7B for which the pretraining corpus is fully released Three findings emerge: a success–locality trade-off where Next-best k=5 achieves 81.5% suppression with zero catastrophic failures on OLMo-7B (with the same edit-mode separation across the other three models); complementarity across edit modes covering 96.5% of OLMo-7B sequences in ensemble; and a hard “mechanism” limit at ∼14% of OLMo-7B sequences where MLP-based editing cannot reach Limitations and Ethics Statement The work aims to mitigate memorization risks, with positive impact via memorization mitigation for privacy/copyright/security, with negative-impact discussion of recoverability under adversarial probing, dual-use risk for factual rewriting, and stakeholder bias from corpus-access requirements The method can be misused for factual rewriting; reliance on pretraining corpus access biases legitimate deployment toward stakeholders with data provenance but misuse is not impossible Open access to data and code Code and mining scripts will be released upon publication under an opensource license All models (SmolLM-360M, OLMo-1B, OLMo-7B, Llama2-7B) and corpora (Dolma v1.7, SlimPajama-6B, smollm-corpus) used are publicly available The paper specifies all hyperparameters (β, k), localization procedure, edit modes, and evaluation protocols required to reproduce the experiments All models (SmolLM-360M, OLMo-1B, OLMo-7B, Llama2-7B) and corpora (Dolma v1.7, SlimPajama-6B, smollm-corpus) used are publicly available The paper specifies all hyperparameters (β, k), localization procedure, edit modes, and evaluation protocols required to reproduce the experiments All models (SmolLM-360M, OLMo-1B, OLMo-7B, Llama2-7B) and corpora (Dolma v1.7, SlimPajama-6B, smollm-corpus) used are publicly available The paper specifies all hyperparameters (β, k), localization procedure, edit modes, and evaluation protocols required to reproduce the experiments All models (SmolLM-360M, OLMo-1B, OLMo-7B, Llama2-7B) and corpora (Dolma v1.

Title and authors: Tom: Now, let’s look at who wrote this. The paper is titled "Output Vector Editing for Memorization Mitigation in Large Language Models," and the authors include Ahmad Dawar Hakimi, Kaiwei Lei, Isabelle Augenstein, Hinrich Schütze >

Jane: It’s a pretty deep team of researchers from places like LMU Munich and the University of Copenhagen working on these kinds of model architectures >

Lu: They’re focusing on how the model stores knowledge in those MLP neurons because they think we need to understand that structure to fix it properly >

Meng: From what I can tell, this approach is focused on finding the specific weight edits needed rather than trying to steer the entire neural network, which is something we've seen before >

Lalam: They are specifically targeting sequences that the model has memorized, like a continuation of a prompt that appeared in its training set >

The paper's summary: Tom: The core finding they highlight is that this method can achieve up to eighty-seven point nine percent suppression on the OLMo-7B model when it’s targeting memorized continuations from its pretraining corpus >

Jane: That eighty-seven point nine percent number is pretty significant, Tom, and the paper explains it by locating those specific MLP neurons responsible for the continuation and minimally modifying their output vectors > <ref:2606.18767#pg3>

Lu: They describe this as a constrained-optimization weight edit that introduces a distractor in vocabulary space by using a rank-one update along a chosen distractor token’s unembedding direction >

Meng: So, they’re not just zeroing things out; they are changing the output vector v i to v'i, which then redirects the residual stream contribution while keeping the activation a i exactly as it was >

Lalam: They also mentioned a process for locating these neurons in two stages, first looking at layer importance and then scoring each neuron by its L1 distance to the post-sequence token >

The paper's improvements: Tom: One of the major practical takeaways they present is that there is a success–locality trade-off across three operating points, like conservative editing with redaction marker at thirty-one point seven percent suppression and near-zero locality cost > <ref:2606.18767#pg3>

Jane: And then they show a practical editing mode, called next-best with k equals five, which achieves eighty-one point five percent suppression while having no catastrophic edits on OLMo-7B >

Lu: They also found that the four edit modes—EOS, redact, next-best, and direct suppression—are complementary; each one succeeds on different sequences where the others might fail >

Meng: But they also identified a mechanistic boundary; approximately fourteen percent of memorized sequences just resist all MLP-based configurations > <ref:2606.18767#pg3,approximately 14% of memorized sequences>

Lalam: And interestingly, they found that if you look at the attention layers, ablating the top contributing attention heads at the target position recovers between sixty and sixty-four percent of those failures > <ref:2606.18767#pg3,ablating the top contributing attention heads at the target position recovers>

Conclusion: Tom: So to wrap up on "Output Vector Editing for Memorization Mitigation in Large Language Models," they show that this weight-level intervention can suppress memorization by modifying what neurons contribute to the residual stream, rather than just whether they are active >

Jane: The implication is that we have a targeted way to reduce privacy and copyright risks without having to overhaul the entire model architecture or training process >

Lu: It really shows how important it is to understand those MLP neurons as knowledge storage because if you know where they are, you can apply this kind of surgical edit >

Meng: For us on the deployment side, the practical editing mode with next-best k equals five seems like the most useful starting point because it gives high suppression without causing massive changes to behavior on OLMo-7B > <ref:2606.18767#pg1>

Lalam: I'm quite confident in this direction; I'd say they have covered a lot of ground by showing this success across four different model sizes and architectures, from 360M up to 7B parameters > <ref:2606.18767#pg1>

Ahmad Dawar Hakimi, Kaiwei Lei, Isabelle Augenstein, Hinrich Schütze

Center for Information and Language Processing, LMU Munich, Germany · Department of Computer Science, University of Copenhagen, Denmark · Munich Center for Machine Learning, Germany · Pioneer Centre for AI, Denmark

cs.CL

Submitted: 2026-06-17

Updated: 2026-10-21

Code: https://github.com/TransformerLensOrg/TransformerLens

License: http://creativecommons.org/licenses/by/4.0/

The gist: The gist The proposed output vector editing method modifies what neurons contribute to the residual stream rather than whether they are active, achieving up to 87.9% suppression on OLMo-7B by

Key concepts

Output Vector Editing: This is a constrained optimization method that targets specific neurons in the model's Multi-Layer Perceptron (MLP). Instead of changing whether a neuron is active, it minimally adjusts its output vector to introduce a 'distractor' token into the vocabulary space. This redirects the residual stream contribution without altering the neuron's original activation state during inference.
Memorization Mining and Localization: The process starts by finding sequences where the model has memorized content from its training data. Localization then identifies which specific neurons are responsible for these memorized continuations. This is done in two steps: first, assessing layer importance based on how much each layer contributes to the target token's logit, and second, scoring neurons based on their distance to the residual stream.
Edit Modes: The method offers four distinct ways to apply the edit, ranging from aggressive suppression (like EOS or Redact) to minimal redirection (like Next-best). These modes are controlled by choosing a specific 'distractor token' ($s$). This allows researchers to balance high suppression rates against the risk of introducing catastrophic edits that degrade overall model performance.

Terminology

Ablation — 1× in this paper · explained in 1 episode
Memorization — 1× in this paper · explained in 3 episodes

Summary

The gist The proposed output vector editing method modifies what neurons contribute to the residual stream rather than whether they are active, achieving up to 87.9% suppression on OLMo-7B by introducing a distractor in vocabulary space while leaving activations unchanged

How it works

The core of the method is output vector editing, a constrained-optimization weight edit that locates a small set of MLP neurons responsible for a memorized continuation and minimally modifies their output vectors to introduce a distractor in vocabulary space, redirecting their residual-stream contributions while leaving activations unchanged This is achieved by modifying the output vector vi of MLP neurons implicated in a memorized continuation The closed-form solution is a rank-one update along a chosen distractor token’s unembedding direction, adapting the rank-one weight-update mechanism of ROME from matrix-level factual editing to perneuron output redirection Unlike activation steering, our edit is computed once per memorized sequence and applied directly to the weights; at inference the model runs unchanged

Memorization Mining and Localization

The process begins with mining memorized content directly from each model’s pretraining corpus using teacher-forced prediction, where outputs agree at the first continuation token, which is the target of our editing For OLMo-7B, 6,831 unique sequences were mined from its pretraining corpus Localization is a two-step process that identifies contributing neurons in two stages Step 1 involves layer importance, where the final residual stream decomposes additively across all layers, and components are projected through the unembedding matrix to calculate an LI value indicating boost or dampening of the target token Step 2 scores each neuron by its L1 distance to r post, where a high score is received if the contribution ai·vi is close to r post in L1 distance Only neurons whose unembedded output vector (vi·WU) ranks the target token either in the top 50 of the vocabulary or in the bottom 50 are selected for editing

Edit Modes and Performance Trade-offs

Four edit modes span a spectrum from aggressive suppression to minimal redirection The modes are defined by the choice of distractor token s, which defines three promotion modes and a fourth mode that dampens the target directly The four edit modes include end-of-sequence (EOS), redaction marker (redact), next-best, and direct suppression Evaluation on OLMo-7B showed a success–locality trade-off across three operating points Conservative editing (redact) at 31.7% suppression has near-zero locality cost, while practical editing (next-best, k=5) at 81.5% has no catastrophic edits

Mechanistic Findings and Limitations

The research identified a mechanistic boundary at ∼14% of sequences unreachable by MLP-only editing Attention-layer analysis showed these failures are not driven by attention overall, but ablating the top contributing attention heads at the target position recovers 56–64% of EOS k=10 failures and 53–60% of Next-best k=5 failures The paper also noted that universal failures, approximately 921 of 6,831 memorized sequences (∼14%) resist every MLP-based configuration These universal failures are often driven by strong sequential dependencies where memorization is distributed redundantly beyond the top-k budget Furthermore, aggressive modes like EOS (k=10) can incur tail risk, with 47.8% of edits having a median shift in perplexity of +0.88

Cross-Model Generalization

Success rates scale with model size rather than family, as SmolLM-360M (34.9%) and OLMo-1B (52.4%) sit well below OLMo-7B's 85.7% success rate Locality costs are architecture-dependent, with the catastrophic-outlier tail on OLMo-7B aggressive modes not appearing on smaller models like OLMo-1B or Llama2-7B The paper concludes that output vector editing is a weight-level intervention that modifies what neurons contribute to the residual stream rather than whether they are active

How it works

The paper presents output vector editing, a neuron-level method for mitigating memorization in LLMs that modifies what neurons contribute to the residual stream rather than whether they are active It was evaluated across four models from two architecture families and different sizes (SmolLM360M, OLMo-1B, OLMo-7B, Llama2-7B), with a deeper grid on OLMo-7B for which the pretraining corpus is fully released Three findings emerge: a success–locality trade-off where Next-best k=5 achieves 81.5% suppression with zero catastrophic failures on OLMo-7B (with the same edit-mode separation across the other three models); complementarity across edit modes covering 96.5% of OLMo-7B sequences in ensemble; and a hard “mechanism” limit at ∼14% of OLMo-7B sequences where MLP-based editing cannot reach

Limitations and Ethics Statement

The work aims to mitigate memorization risks, with positive impact via memorization mitigation for privacy/copyright/security, with negative-impact discussion of recoverability under adversarial probing, dual-use risk for factual rewriting, and stakeholder bias from corpus-access requirements The method can be misused for factual rewriting; reliance on pretraining corpus access biases legitimate deployment toward stakeholders with data provenance but misuse is not impossible

Open access to data and code

Code and mining scripts will be released upon publication under an opensource license All models (SmolLM-360M, OLMo-1B, OLMo-7B, Llama2-7B) and corpora (Dolma v1.7, SlimPajama-6B, smollm-corpus) used are publicly available The paper specifies all hyperparameters (β, k), localization procedure, edit modes, and evaluation protocols required to reproduce the experiments All models (SmolLM-360M, OLMo-1B, OLMo-7B, Llama2-7B) and corpora (Dolma v1.7, SlimPajama-6B, smollm-corpus) used are publicly available The paper specifies all hyperparameters (β, k), localization procedure, edit modes, and evaluation protocols required to reproduce the experiments All models (SmolLM-360M, OLMo-1B, OLMo-7B, Llama2-7B) and corpora (Dolma v1.7, SlimPajama-6B, smollm-corpus) used are publicly available The paper specifies all hyperparameters (β, k), localization procedure, edit modes, and evaluation protocols required to reproduce the experiments All models (SmolLM-360M, OLMo-1B, OLMo-7B, Llama2-7B) and corpora (Dolma v1.

Improvements for AI systems

Output vector editing can be applied to large language models to suppress memorized sequences by modifying only a small set of MLP neurons responsible for a memorized continuation, achieving up to 87.9% suppression on OLMo-7B.
The system can implement four distinct edit modes—EOS, Redact, Next-best, and Suppress—allowing for a spectrum from aggressive suppression to minimal redirection based on the chosen distractor token s.
The method can locate responsible neurons in two steps: first selecting important MLP layers based on their contribution to the target token's logit, and second scoring individual neurons by their L1 distance to the residual stream.
The system can use a closed-form solution, v'i = vi + ∆i − u T s vi / us squared, which requires only the unembedding matrix and no gradient computation for applying the rank-one weight update at inference time.
The model can be configured to prioritize practical editing by using Next-best (k=5) mode, achieving 81.5% suppression with zero catastrophic edits and a maximum median perplexity shift of +0.13 on OLMo-7B sequences.
The system can incorporate an attention-layer analysis as a complementary fallback mechanism to identify failures that are not purely MLP-driven, recovering 56–64% of EOS k=10 failures by ablating the top contributing attention heads at the target position.
The system can be designed with a skip filter that avoids editing ubiquitous tokens like stopwords or punctuation during localization to prevent disproportionate effect on locality, ensuring edits are sequence-specific.

Abstract

Large language models can reproduce training text, creating privacy, copyright, and security risks. We propose Output Vector Editing (OVE), a post-hoc method for suppressing text continuations that a model reproduces from a given prefix. Under superposition, a neuron can contribute to multiple features. Yet some neuron-level mitigation methods suppress localized neurons by zeroing their activations, removing each neuron's entire contribution. OVE instead edits their output vectors, combining (i) two-stage localization of MLP layers and neurons with (ii) full-vocabulary least-squares updates. Its multi-anchor variant, OVE-MA, adaptively selects continuation positions expected to provide stronger suppression relative to the required weight change. We evaluate recovery and locality across four models (360M--7B parameters), using autoregressively validated continuations and recovery tests with greedy decoding, beam search, and sampling. On 634 OLMo-7B pairs, OVE-MA with at most five output-vector updates per request leaves 24.45% recoverable in these tests, compared with 32.02% for the closest prior post-hoc editing method, REVS, while using approximately 75 times fewer update operations. These results show that adaptive position selection can concentrate effective suppression into a small number of weight updates. OVE edits weights without gradients and requires no additional intervention during inference.