On-Policy Distillation with Negative-Policy Rollouts

The gist The introduction introduces Negative-Policy Distillation (NP-OPD), which complements teacher supervision with rollouts from a lower-performing, lower-capability negative policy to provide an explicit negative signal for the student model during on-policy distillation.

On-Policy Distillation and its Limitations

On-policy distillation (OPD) trains a student model using trajectories generated by the student itself and token-level supervision from a stronger teacher model The teacher serves as the only positive role model in distillation When the student’s distribution sufficiently overlaps with that of the teacher, the positive-only guidance provides enough signal for on-policy distillation However, limited overlap between the teacher and student distributions provides insufficient supervision for OPD

Negative-Policy Distillation (NP-OPD) Mechanism

NP-OPD introduces a negative policy into an on-policy distillation framework by modifying the rollout distribution to include both student-generated trajectories and trajectories sampled from an additional negative policy, πn Each training batch consists of trajectories sampled from the student policy and the negative policy, with proportions 1 − α and α, respectively The resulting RL-style update direction is written as ∇θJNP(θ) = Ex∼D, zx∼Bernoulli(α), y∼πzx (·x) [X y t=1 ROPD t ∇θ log πθ(yt x, y<t)] This formulation changes the trajectory distribution exposed to teacher supervision, while the token-level teacher-student supervision remains unchanged

Mathematical Justification for Negative Signals

The difference between the gradient contributions of OPD and NP-OPD quantifies the gradient changes induced by sampling from the negative policy The difference is given by gNP − gOPD = α "Yt i=1 πn(yi x, y<i) − Yt i=1 πθ(yi x, y DKL(πn∥πθ), the expected value of OPD reward in Eq. 2 is presented as Eyt∼πn ROPD t = Eyt∼πn [log π∗ (yt) − log πθ(yt)] = -DKL(πn∥π∗) + DKL(πn∥πθ) < 0 In short, the tokens generated by the negative policy have negative expected OPD rewards This suggests that the OPD reward trains the student not to generate tokens from the negative policy

Empirical Validation and Effectiveness

Extensive experiments show that NP-OPD consistently improves OPD across model scales, generation modes, reasoning domains, and different OPD variants NP-OPD effectively suppresses tokens preferred by the negative policy over the teacher and moves the student away from the negative policy The analysis shows that NP-OPD concentrates suppression on negative tokens and reduces overlap with the negative policy Furthermore, NP-OPD can reuse pre-generated negative-policy rollouts, reducing the cost of rollout generation during training

Analysis of Suppression and Overlap Reduction

The negative-policy suppression precision (NSP) is defined as the fraction of suppressed tokens that are preferred by the negative policy over the teacher NP-OPD using a lower-capability rollout policy than the student yields higher NSP and higher reasoning performance The negative-policy overlap reduction (NOR) measures a shift in the student’s probability gap toward the teacher’s top prediction relative to the negative policy’s at disagreement positions NP-OPD results show a mean NOR of 0.019 under NP-OPD compared with 0.005 under OPD, indicating a greater relative shift toward the teacher

Conclusion on Complementary Signals

NP-OPD can complement the implicit negative signals in recent OPD methods such as ExOPD and OPD2 The negative signal introduced through negative-policy rollouts can complement the implicit negative signals in these OPD methods without modifying their reward functions NP-OPD is supported not only by empirical performance gains but also by direct analyses of the effects induced by negative-policy rollouts

Practical Advantages and Synthesis

Training efficiency is a practical advantage of NP-OPD because negative-policy rollouts can be pre-generated and reused throughout training For standard OPD, using only negative-policy rollouts (α = 1) reduces the training time from 9h19m to 3h34m in thinking mode and from 5h15m to 1h16m in non-thinking mode Model merging can combine some of the benefits of using on-policy rollouts and negative rollouts This suggests that simple degradation of the student can partially synthesize useful negative rollouts, but does not match the benefit of using a separate lower-capability policy to provide a move-away direction in OPD

Further Research Directions

The paper examines whether similar negative rollouts can be synthesized from the student by increasing the decoding temperature to 1.5, adding a persona prompt that asks it to solve the problem at an elementary-school level, or dropping three layers during rollout generation These modifications improve over standard on-policy rollouts but do not match the benefit of using a separate lower-capability policy to provide a move-away direction in OPD The analysis also shows that degrading student rollouts improves OPD but is less effective than using a separate negative policy

Model Merging Insights

Model merging for Qwen3-8B in thinking mode, motivated by the observation that the best rollout mixing ratio varies across domains, merges models trained with α = 0 and α = 1 by averaging their parameters with equal weights These results suggest that model merging can combine some of the benefits of using on-policy rollouts and negative rollouts Model merging with ExOPD and OPD2 shows improvements over the on-policy baseline from 74.2 to 76.3 on Math, from 60.3 to 60.7 on Code, and from 53.6 to 54.7 on Science

Benchmark Performance Across Variants

NP-OPD improves overall average performance across all three formulations for Qwen3-1.

1. What is improved is On-Policy Distillation (OPD) by introducing Negative-Policy OPD (NP-OPD), which complements teacher supervision with rollouts from a lowerperforming, lower-capability negative policy that serves as a negative reference for the student. This allows the student to learn away toward undesirable outputs by being exposed to tokens preferred by the negative policy over the teacher, while preserving the original distillation reward direction.

2. The improved system can suppress undesirable generations more effectively than standard OPD, as NP-OPD results show that it suppresses tokens preferred by the negative policy over the teacher and moves the student away from the negative policy. This targeted suppression is quantified by a metric called Negative-Policy Suppression Precision (NSP), which measures the fraction of suppressed tokens that are preferred by the negative policy over the teacher.

3. The system can generalize its learning to unseen data, as analyses show that NP-OPD's effect is not limited to the prompts used during training and generalizes to unseen benchmark examples. This means the model maintains a greater relative shift toward the teacher even on held-out prompts, indicating robust learning beyond the training set.

4. Training efficiency can be enhanced by reusing pre-generated negative rollouts, which is noted as a practical advantage of NP-OPD because these rollouts can be pre-generated and reused throughout training. This significantly reduces the cost of generating on-policy rollouts during the actual training process.

5. The system can leverage different OPD reward formulations without modification, as NP-OPD can be combined with recent advances in OPD to further improve their performance and is effective across different OPD variants, including ExOPD and OPD2. This suggests that the negative signal complements implicit negative signals already present in these advanced distillation methods.

6. Model behavior can be steered toward specific reasoning styles, such as an elementary school level, by using a persona prompt to instruct the student model to Think and write the way a young student would. This demonstrates that NP-OPD can be combined with other techniques like persona prompting to achieve targeted behavioral modifications.

7. The system can adapt its internal structure for better performance by applying layer dropping, where we remove the last three transformer layers from the student model in a layer-drop setting. This suggests that structural modifications, when paired with negative policy rollouts, can be effective in improving distillation outcomes.

On-policy distillation (OPD) has been widely studied as a post-training method in which a student model obtains token-level supervision from a stronger teacher on its own rollouts. Recent studies have improved OPD through alternative distillation reward formulations and teacher configurations, while the objective of distillation remains centered on mimicking the teacher. However, when a stronger teacher has limited distributional overlap with the student, such positive guidance can provide insufficient learning signals. In this work, we introduce Negative-Policy OPD (NP-OPD), which complements teacher supervision with rollouts from a lower-performing, lower-capability negative policy that serves as a negative reference for the student. Rather than modifying the distillation reward formulation, NP-OPD introduces the negative policy at the rollout stage, continuously supplying tokens preferred by the negative policy over the teacher so that they remain exposed to teacher supervision throughout training. This provides an explicit negative signal through negative-policy rollouts while preserving the positive teacher supervision used in OPD. Through extensive experiments, we show that NP-OPD improves OPD across model scales, generation modes, reasoning domains, and different OPD variants. Furthermore, our analyses show that NP-OPD effectively suppresses tokens preferred by the negative policy over the teacher and moves the student away from the negative policy. These results support our design of introducing negative signals through negative-policy rollouts and provide new insight into the role of the rollout policy in OPD. Code will be available at https://github.com/naver-ai/np-opd.