Recovering Reliable Supervision via Critique-Guided On-Policy Distillation
cs.LG, cs.AI
Submitted: 2026-10-14
Updated: 2026-10-14
Comments: 21 pages. Code: https://github.com/kzhao5/CriticOPD
Code: https://github.com/kzhao5/CriticOPD
License: http://creativecommons.org/licenses/by/4.0/
The gist: On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student
Terminology
Abstract
On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes. We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them. We propose CriticOPD, using error recognition to restore recoverability. The teacher critiques an incorrect rollout, allowing the student to rewrite its solution following the error. The resulting output, including the critique as privileged information, is scored by the teacher while the student uses the revision without the critique for training. On two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD improves OPD, with the largest gains on competition problems.
Sources
- A General Language Assistant as a Laboratory for Alignment
- MathArena: Evaluating LLMs on Uncontaminated Math Competitions
- Trajectory-Refined Distillation
- SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision
- Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
- Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation
- Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation
- Training Language Models with Language Feedback at Scale
- Learning by Distilling Context
- Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation
- Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation
- Escaping the KL Agreement Trap in On-Policy Distillation
- Pass the Baton: Trajectory-Relayed On-Policy Distillation
- OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning
- When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation
- Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning
- SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting
- Less is More: Early Stopping Rollout for On-Policy Distillation
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks