Defense via Consensus: Mitigating Subliminal Learning through Multi-Teacher Distillation

arXiv:2607.23394 · cs.AI · Submitted 2026-07-25 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Defense via Consensus".

Jane: The gist: Consensus decoding suppresses source-specific misbehavior while preserving shared desirable behavior across multiple datasets by aggregating next-token distributions from separately fine-tuned reference models.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we’re starting with the title and authors of this paper. It’s "Defense via Consensus: Mitigating Subliminal Learning through Multi-Teacher Distillation" by Adhyyan Narang, Artin Tajdini, and Claire Zhang >

Jane: The authors are researchers at a University of Washington and they’re tackling something called subliminal learning attacks, which is when poisoned data tries to teach the model things without leaving an obvious signature in the input data itself >

Lu: The core idea they are proposing is using inference-time consensus for mitigating hidden behaviors from LLM fine-tuning >

Meng: So they are not trying to fix the training process directly, but rather they fine-tune a separate reference model on each source’s dataset and aggregate their next-token distributions at decoding time >

Lalam: The paper is exploring whether collecting multiple datasets from different sources for the same task can help us prevent vulnerability to malicious actors >

The paper's summary: Tom: So what they are summarizing is that standard defenses like data filtering or mixing in harmless data don't eliminate these hidden preferences, and even KL regularization toward a collection of reference models averages their distributions so behavior carried by one source is diluted but never vetoed >

Jane: They show that the structural asymmetry between the desired and poisoned behavior is captured by assumption A2, which means the desired behavior is shared across more sources than any single cost >

Lu: The paper argues that a shared cost would require aligned attack objectives and coordination overhead, making it less likely in practice >

Meng: They show that fine-tuning one reference model per source and keeping only what their next-token distributions agree on removes the source-specific behavior >

Lalam: Basically, the defense rests on the fact that no single source can lift a token’s probability on its own, so the aggregate stays safe unless a majority of sources teaches the same misbehavior >

The paper's improvements: Tom: They introduce two specific consensus decoders for aggregation during decoding, which are called a token-wise minimum and a base-relative variant >

Jane: The token-wise minimum caps each token at the lowest probability any source assigns, while the base-relative variant reverts to the base probability on any token where sources move in opposing directions >

Lu: These decoders are compared against union training, which trains one adapter on all pooled source data, and LoRA weight merging, which averages the source adapter updates >

Meng: The paper validates this across three settings: explicit prefix poisoning, subliminal learning, and emergent misalignment >

Lalam: In the explicit prefix poisoning setting, they found that both union training and weight averaging retain both the shared joke behavior and the private prefixes >

Conclusion: Tom: So to wrap up on "Defense via Consensus: Mitigating Subliminal Learning through Multi-Teacher Distillation," they’re showing that consensus decoding suppresses source-specific misbehavior while preserving shared desirable behavior >

Jane: They confirm that even union training and weight averaging can retain the unwanted behavior, but their consensus decoders are more effective at separating the two at decoding time >

Lu: The token-wise minimum generally retains more of the shared behavior, and base-relative minimum suppresses changes that don't agree across sources more strongly >

Meng: They also extended this idea to handle partial support with quorum aggregation, where a behavior is retained if it’s supported by strictly more sources than any single cost >

Lalam: And they even introduced semantic smoothing to handle surface-form disagreement by embedding continuations and comparing them using cosine similarity to generate pseudo-support before token-level quorum aggregation >

Tom: So the conclusion is that consensus measures support, not correctness, and that when useful behavior repeats across sources but unwanted behavior remains source-specific, consensus can separate those two at decoding time >

University of Washington

cs.AI

Submitted: 2026-07-25

Updated: 2026-10-17

Comments: 42 pages, 8 figures. Code: https://github.com/AdhyyanNarang/consensus-aggregation

Code: https://github.com/AdhyyanNarang/consensus-aggregation

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: The gist: Consensus decoding suppresses source-specific misbehavior while preserving shared desirable behavior across multiple datasets by aggregating next-token distributions from separately

Key concepts

Consensus Decoding: A method during generation where the final output token is chosen based on a consensus among multiple source models. This process aims to filter out errors or malicious outputs specific to one source by requiring agreement from several sources, thereby preserving the general, shared correct behavior.
Token-wise Minimum Decoder: One of two proposed decoding strategies where each individual token is capped at the lowest probability assigned by any of the source models. This mechanism aggressively suppresses tokens that are highly likely to be specific errors or poisoned outputs from a single source.
Base-Relative Variant: A second consensus decoder that focuses on changes relative to a common base model. It keeps the smallest change when all sources agree on a direction but reverts to the base probability if sources disagree, helping maintain fidelity to the original, desired behavior.
Quorum Aggregation: An extension of consensus that replaces strict unanimity with a support threshold. A token is accepted if it receives support from more than a predefined number of sources, ensuring robustness even when some sources provide conflicting or weak signals.

Terminology

Decoder — 1× in this paper · explained in 2 episodes
Distillation — 1× in this paper · explained in 2 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes
Redundancy — 1× in this paper · explained in 2 episodes
Regularization — 1× in this paper · explained in 3 episodes
Robustness — 1× in this paper · explained in 5 episodes

Summary

The gist: Consensus decoding suppresses source-specific misbehavior while preserving shared desirable behavior across multiple datasets by aggregating next-token distributions from separately fine-tuned reference models.

Problem Formulation

The goal is to construct a learning method that preserves a desired benefit while suppressing source-specific costs for settings where the benefit is shared and sources are distinguishable The learner sees which dataset came from which source, but the benign/adversarial identity of each source is hidden The desired behavior is shared; poisoned behaviors are source-specific.

Threat Model and Contributions

The defense rests on the structural asymmetry between the desired and the poisoned behavior, captured by Assumption (A2), where the desired behavior is supported by strictly more sources than any single cost. The main goal of our paper is to construct a learning method that preserves the benefit b while suppressing the costs ci for settings that satisfy these assumptions. Our contributions include demonstrating that consensus decoding suppresses source-specific misbehavior while preserving shared desirable behavior, including cases where union training and weight averaging retain the unwanted behavior, and introducing "two consensus decoders: a token-wise minimum, which caps each token at the lowest probability any source assigns, and a base-relative variant, which reverts to the base probability on any token the sources move in opposing directions".

Consensus Aggregation Methods

The paper introduces two specific consensus decoders for aggregation during decoding. Minimum aggregation operates on absolute probability mass, where the raw score smin(v c) is defined as min i πi(v c), πmin(v c) = smin(v c) / P u∈V smin(u c). Base-relative minimum instead operates on changes from the common base model, keeping the smallest-magnitude change when every source agrees on its direction and reverting to the base score otherwise. These methods are compared against union training πunion, which trains one adapter on the pooled source data; LoRA weight merging πmerge, which averages the source adapter updates.

Empirical Validation Across Tasks

The method is validated across three controlled settings: explicit poisoning, subliminal learning, and emergent misalignment. In the explicit prefix poisoning setting, πmerge and πunion retain both the shared joke behavior and the private prefixes. In the subliminal learning setting, the reference models learn the joke behavior, but each also changes the animal probe associated with its source. In emergent misalignment, πmin and π∆ reduce both broad and medical failures while preserving most of the shared benefit.

Relaxed Agreement Mechanisms

The paper extends exact agreement with two mechanisms to tolerate partial support or different surface expressions of the same intention. Quorum aggregation replaces unanimity with a support threshold, where Φq(v c) ∝ q-th largest of πi(v c) m i=1. Semantic smoothing is introduced to handle surface form disagreement by embedding continuations and comparing them using cosine similarity to generate pseudo-support that is added to each reference model’s current distribution before token-level quorum aggregation.

Conclusion on Robustness

The experiments support the conclusion that when useful behavior repeats across sources and unwanted behavior remains source-specific, consensus can separate the two at decoding time. Minimum aggregation generally retains more of the shared behavior, while base-relative minimum more strongly suppresses changes that do not agree across sources. Quorum and semantic smoothing extend the same idea when agreement is incomplete or expressed with different tokens. The paper concludes by noting that Consensus measures support, not correctness.

Improvements for AI systems

This system can be made robust against source-specific misbehavior by employing consensus decoding, which suppresses source-specific misbehavior while preserving shared desirable behavior across controlled poisoning tasks and emergent misalignment.
The system utilizes two consensus decoders: a token-wise minimum, which caps each token at the lowest probability any source assigns, and a base-relative variant, which reverts to the base probability on any token the sources move in opposing directions.
The system can be extended to handle partial support using Quorum aggregation, where a behavior is retained if it is supported by strictly more sources than any single cost, defined by assumption (A2).
Semantic smoothing can be integrated during decoding to handle surface-form disagreement, as it allows the system to compare continuations using embeddings and transfer pseudo-support before token-level quorum aggregation.
The resulting AI system will retain shared benefits while reducing source-specific costs, as demonstrated by results showing that consensus decoders preserve shared behavior while reducing unwanted behavior tied to one source.

Abstract

Fine-tuning can introduce unwanted behaviors through deliberate poisoning or unexpected generalization from seemingly benign data. Because these effects may be difficult to anticipate or detect, we seek robustness in the fine-tuning process itself. Our premise is that datasets independently collected for the same task would share the intended capability while differing in their unwanted effects. This creates an opportunity to learn from agreement across sources while reducing dependence on any individual source. We introduce Multi-Source Consensus Distillation (MSCD): we fine-tune one model on each dataset, use agreement among these models to generate replacement training responses, and train a fresh student on the resulting data. We develop minimum and base-relative rules for combining the models' predictions, with extensions that accommodate useful behaviors learned by only some models or expressed in different words. Experiments on controlled poisoning, subliminal learning, and emergent misalignment show suppression of unwanted behavior while retaining shared benefits. In a practical language-understanding task, MSCD retains intent-classification capability while reducing unsafe medical advice introduced during fine-tuning. Comparing distilled students with direct consensus decoding shows that distillation closely preserves the measured desirable and undesirable behavior rates while replacing multiple model evaluations per generated token with a single student evaluation.