Demystifying On-Policy Distillation: Where Does the Teacher's Advantage Live?
cs.LG
Submitted: 2026-10-13
Updated: 2026-10-13
License: http://creativecommons.org/licenses/by/4.0/
The gist: On-policy distillation (OPD) is increasingly studied as a cheaper alternative to reinforcement learning, yet it remains a form of knowledge distillation (KD) that relies on teacher supervision.
Terminology
Abstract
On-policy distillation (OPD) is increasingly studied as a cheaper alternative to reinforcement learning, yet it remains a form of knowledge distillation (KD) that relies on teacher supervision. We revisit OPD from the KD perspective by asking where the teacher's advantage lies along student-generated trajectories. Controlled token flips at teacher-student disagreements directly measure the accuracy gained from teacher corrections. Single-token flips yield a modest average gain on selected failed trajectories, whereas repeated flips raise accuracy from 28.4% to 74.5%, close to the teacher's 76.9%. Gains continue to accumulate with more flips, but mass- and KL-based selection outperform random flips by 15-18 percentage points at counts of similar magnitude. Selective intervention remains useful after OPD, but greater argmax agreement does not consistently improve accuracy across model pairs. We advocate training schedules that learn more valuable teacher corrections while maintaining distributional stability. Controlled flips and fork-resolution analysis evaluate OPD beyond accuracy by measuring which teacher preferences the student acquires and what further correction can recover.
Sources
- On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
- Accelerating Large Language Model Decoding with Speculative Sampling
- ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes
- GLM-5: from Vibe Coding to Agentic Engineering
- Distilling the Knowledge in a Neural Network
- Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe
- Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level
- Trajectory-Refined Distillation
- Entropy-Aware On-Policy Distillation of Language Models
- Kimi K3: Open Frontier Intelligence
- Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
- Tuning Language Models by Proxy
- Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation
- Simple-OPD: Demystifying Warm-up for On-policy Distillation
- Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergence
- Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
- KL for a KL: On-Policy Distillation with Control Variate Baseline
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks