Privileged Context as Drift in On-Policy Self-Distillation

arXiv:2610.07842 · cs.LG · Submitted 2026-10-06 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Privileged Context as Drift in On-Policy Self-Distillation".

Jane: The gist The choice of privileged context affects policy drift in on-policy self-distillation,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, wrapping up, this paper "Privileged Context as Drift in On-Policy Self-Distillation" shows that the way you pick your privileged context—whether it’s a demonstration, feedback, or a rephrase—is a critical factor in policy drift during on-policy self-distillation.

Jane: It emphasizes that simply picking the right context for accuracy isn't enough; you need to understand how that context choice dictates the actual movement of the model away from its starting point.

Lu: The main implication is that for continual learning, privileged context shouldn't just be seen as input data; it needs to be part of the stability design because it controls the direction and magnitude of policy changes <ref:2610.07842#pg2>.

Meng: From a deployment standpoint, this means we can’t just assume that whatever context we feed in is benign; we need to consider its structure and source when designing our distillation process <ref:2610.07842#pg3>.

Tom: Exactly. They found that content choices span a wider range of distributional drift than source choices, which means you really need to be careful how you define your context inputs for stability <ref:2610.07842#pg1>.

Jane: So for listeners who aren't deep in the math, it’s about recognizing that the extra information you give an AI model—the privileged context—actually steers its learning path in measurable ways <ref:2610.07842#pg3>.

Conclusion: Tom: So, we’ve been looking at this paper called "Privileged Context as Drift in On-Policy Self-Distillation" and I gotta say, the setup they used is really specific.

Jane: It seems like the core idea revolves around how picking that privileged context—whether it's a demonstration or some feedback—actually changes how much the AI model drifts away from its original training.

Lu: Yeah, they’re testing this by varying two things: what's in the context, like a rephrase or a demonstration, and where that context comes from, like an external source or one generated by the AI itself with some verification.

Meng: From my side on production stuff, I’m curious about how much of that drift is actually useful versus just making the model wander off track.

Lalam: Based on what I've processed, the authors suggest that we need to start treating this context choice as a part of our stability design in self-distillation because it’s directly tied to how far and in what direction the policy moves.

Tom: That’s a strong statement. It sounds like they aren't just looking at accuracy gains, but more at the actual mechanics of model update geometry.

Jane: Exactly, Tom. They measured things like target-task accuracy, retention on old tasks, and even how much the parameter updates align with each other after a distillation step.

Lu: And what they found is that content choices create a wider range of drift compared to just changing the source of the context. The ratio between those ranges is pretty significant for both token level and sequence level KL divergence.

Meng: That means if we're using this for continual learning, we can't just pick whatever context gives us the highest accuracy on one task; we have to consider how that specific context steers the AI’s overall movement.

Tom: It puts a lot of pressure on us to design our distillation protocols with stability in mind rather than just chasing the highest single benchmark score.

Jane: And it opens up a bigger question about what kind of privileged context generators we should be looking for if we want truly reliable self-distillation systems.

Ravenor Davion, Nick Rui

Stanford University

cs.LG

Submitted: 2026-10-06

Updated: 2026-10-06

Comments: 18 pages, 4 figures, 5 tables

Code: https://github.com/huggingface/Math-Verify

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: The gist The choice of privileged context affects policy drift in on-policy self-distillation, and these findings suggest that privileged context should be treated as part of OPSD’s stability

Key concepts

On-Policy Self-Distillation (OPSD): A training method where a language model learns to match a copy of itself, conditioned on special 'privileged context.' This allows the model to improve by mimicking its own outputs based on specific examples or instructions, rather than just general data.
Privileged Context Content: This refers to what is used as the privileged context, which can be a demonstration (showing an explicit solution), feedback (diagnostics on a mistake), or a rephrase (restating the query). The study tests how these different types of content influence how much the model's policy drifts away from its original state.
Policy Drift: This measures how much the language model's behavior changes over time during training. In this context, it tracks the distributional drift (how different the model's outputs are) and parameter-update geometry (how closely related the model's internal changes are to previous versions).
Source Type: This describes where the privileged context comes from. It can be external data, self-generated content created with a verifier, or self-generated content without a verifier. The research compares how the origin of this context impacts the stability and direction of policy updates.

Terminology

Distillation — 1× in this paper · explained in 2 episodes
Self-Distillation — 1× in this paper · explained in 7 episodes

Summary

The gist The choice of privileged context affects policy drift in on-policy self-distillation, and these findings suggest that privileged context should be treated as part of OPSD’s stability design because it is associated with how far and in what direction the policy moves

How it works

On-policy self-distillation (OPSD) trains a language model to match a copy of itself conditioned on privileged context Existing work varies what privileged context contains and how it is produced while also changing models, data, and training setups, making the effects of privileged context design difficult to isolate. The study varies two axes: content (a demonstration, feedback, or rephrase) and source (external, selfgenerated with a verifier, or self-generated without a verifier). They train Qwen2.5-7B with OPSD across these nine combinations and three datasets.

Privileged Context Matrix

The researchers evaluate a full 3×3 content-by-source grid on three domains of tasks. They measure drift using per-token KL divergence from the base policy, cosine similarity between parameter updates, and changes in accuracy on prior-task benchmarks. Table 1 summarizes the repository implementation of every cell. The content types studied are demonstrations, which provide an explicit solution to the query; feedback, which provides diagnostics on a previous attempt; and rephrases, which restate the query without providing an answer.

Evaluation Metrics and Findings

The evaluation measures target-task accuracy, prior-task retention, distributional drift, and LoRA weight geometry. The study found that content choices span a wider range of distributional drift from the base policy than source choices. Specifically, holding source fixed, changing content spans a wider median KL range than holding content fixed and changing source, with ratios of 5.1× for per-token KL and 2.2× for per-sequence KL. Parameterupdate geometry showed that updates from adapters that share content are more closely aligned (mean cosine 0.571) than updates from adapters that share source (0.255).

Implications for Continual Learning

The findings suggest that privileged context should be treated as part of OPSD’s stability design because it is associated with how far and in what direction the policy moves. The paper concludes that Evaluating acquisition, retention, and drift separately gives a more complete account of whether an OPSD update adds a capability while limiting movement from the base policy.

Limitations

Several limits bound this interpretation. These include that the content and source levels differ in kind, so the held-fixed KL comparisons depend on the selected levels, and their magnitude depends on whether divergence is aggregated per token or per sequence. Furthermore, because response length varies with content type, neither summary is length-invariant. The External source confounds authorship with model identity and capacity, so source differences cannot be attributed to generator capacity or verification individually.

The paper concludes that OPSD systems intended for continual learning should evaluate acquisition, retention, and drift separately rather than select privileged context using target-task accuracy alone. The study remains open which properties of privileged context generators account for the source-associated differences observed here. The paper is available as arXiv:2610.07842v1 [cs.LG] 6 Oct 2026

--- Page 35 ---

Target-task accuracy 35% 45% 55% 65% 75% 85%

--- Page 10 ---

The paper is available as arXiv:2610.07842v1 [cs.LG] 6 Oct 2026

--- Page 5 ---

Figure 3 Target-task and prior-task accuracy by dataset and privileged context condition

--- Page 8 ---

Figure 2 Target-task accuracy change and per-token distributional drift by privileged context condition

--- Page 17 ---

Table 3 Mean prior-task accuracy change by dataset and privileged context condition

--- Page 20 ---

Table 5 Task accuracy, distributional drift, response length, update norm, and effective rank by privileged context condition

--- Page 18 ---

Table 5 Task accuracy change by benchmark and privileged context condition

--- Page 26 ---

References Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations (ICLR), 2024

References Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, and Nick Haber. Big-Math: A largescale high quality math dataset for reinforcement learning in language models. arXiv preprint arXiv:2502.17387 2025

References Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph

References DeepSeek-AI DeepSeek-V4: Towards highly efficient million token context intelligence. arXiv preprint arXiv:2606.19348 2026

References Leichao Dong, Dongxu Zhang, Yiding Sun, Qirui Wang, Yuhan Wang, Lin Chen, and Jihua Zhu. Better starts better ends: Bootstrapped iterative self-reasoning distillation for compressed reasoning. arXiv preprint arXiv:2607.15736 2026

References Yu Fu Longxuan Yu Haz Sameen Shahgir, Zhipeng Wei, Hui Liu, N. Benjamin Erichson, and Yue Dong. Reducing the safety tax in LLM safety alignment with on-policy self-distillation. arXiv preprint arXiv:2605.15239 2026

References Leo Gao Jonathan Tow Baber Abbasi Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h

References John Gkountouras Josip Jukic and Ivan Titov. Consensus as privileged context for label-free self-distillation. arXiv preprint arXiv:2607.13643 2026

References Siyi Gu Jialin Chen Sophia Zhou Arman Cohan, Rex Ying. Rethinking reward supervision: Rubric-conditioned self-distillation. arXiv preprint arXiv:2606.19327 2026

References Dan Hendrycks Collin Burns Steven Basart Andy Zou Mantas Mazeika Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), 2021

References Edward J Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models.

Improvements for AI systems

Bold header: Continuous Stability Design Integration

The improved system should treat privileged context as part of OPSD’s stability design because it is associated with how far and in what direction the policy moves. This means OPSD training prefixes should be conditioned on a fixed teacher and the selected privileged context, ensuring that the initialized teacher anchors training targets to the base policy, while privileged context supplies the initial teacher–student disagreement.

Bold header: Content vs. Source Sensitivity Tuning

The system can dynamically select content and source based on desired drift characteristics; for instance, Holding source fixed, changing content spans a wider median KL range than holding content fixed and changing source, allowing researchers to deliberately choose between large distributional shifts (content) or more constrained updates (source).

Bold header: Separate Evaluation Metrics for Capability Acquisition

Instead of relying solely on target-task accuracy, the system should evaluate acquisition, retention, and drift separately to gain a more complete account of whether an OPSD update adds a capability while limiting movement from the base policy.

Abstract

On-policy self-distillation (OPSD) trains a language model to match a copy of itself conditioned on privileged context. Existing work varies what privileged context contains and how it is produced while also changing models, data, and training setups, making the effects of privileged context design difficult to isolate. Motivated by efforts in continual learning to reduce catastrophic forgetting, we study how the choice of privileged context affects policy drift. Specifically, we vary two axes: content (a demonstration, feedback, or rephrase) and source (external, self-generated with a verifier, or self-generated without a verifier). We train Qwen2.5-7B with OPSD across these nine combinations and three datasets, measuring target-task accuracy, prior-task retention, reverse KL from the base policy, and parameter-update geometry. Holding source fixed, changing content spans a wider median KL range than holding content fixed and changing source. The ratio between these ranges is 5.1 times for per-token KL and 2.2 times for per-sequence KL. Parameter-update geometry shows the same pattern: updates from adapters that share content are more closely aligned (mean cosine 0.571) than updates from adapters that share source (0.255). For continual learning, these findings suggest that privileged context should be treated as part of OPSD's stability design because it is associated with how far and in what direction the policy moves.