Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimization

arXiv:2606.07000 · cs.AI · Submitted 2026-06-05 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Teaching the Way, Not the Answer".

Jane: Here's the gist of the paper. The gist The proposed PTD-PO framework provides dense token-level guidance for failed rollouts by conditioning a self-teacher on non-answer-revealing spatial and textual hints, while keeping the student under the original answer-free context.

Tom: Reinforcement Learning with Verifiable Rewards (RLVR) often suffers from sparse verifiable rewards that provide little token-level supervision for failed rollouts, leading to inefficient exploration in complex multimodal reasoning tasks 5, "the sparse nature of verifiable rewards provides little token-level supervision for failed rollouts, often leading to inefficient exploration in complex multimodal reasoning tasks" 1>.

Jane: Online Policy Distillation (OPD) requires substantial computational overhead and may introduce tokenizer or vocabulary mismatch when using external teachers 2, "Online Policy Distillation (OPD) provides such guidance by aligning student rollouts with a teacher policy during training" 4>.

Tom: Existing self-distillation methods often use ground-truth answers or complete solutions as conditioning signals, which may induce answer-level shortcuts, resulting in "overly deterministic trajectories, sharpened token distributions, accelerated entropy collapse, and reduced exploration of alternative reasoning paths" 5>.

Title and authors: Tom: Let's go over the summary of "Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimization" again. They’re essentially saying PTD-PO is a framework for RLVR that offers dense guidance without exposing the answer to the student policy >

Jane: It focuses on identifying that sparse outcome rewards and solution-revealing self-distillation are limiting exploration in multimodal reinforcement learning tasks >

Lu: So they introduce this answer-free privileged tutoring framework which routes dense supervision to failed trajectories by conditioning a frozen reference model on spatial and textual hints, while keeping the student under the original answer-free context >

Meng: The key is that these hints are generated offline by prompting a strong model to reverse engineer a guidance signal from an answer consistent but non-revealing, so it’s not just random noise >

Tom: And they also propose the PTD-PO objective, which combines the GRPO loss with the PTD regularizer to achieve this dense supervision on failed rollouts >

Jane: It’s designed to complement sparse outcome-level RLVR with dense token-level guidance, similar to how online policy distillation works but more efficiently >

Lu: They show that this approach consistently improves over RLVR and distillation-based baselines when evaluated across LVLMs from 2B up to 8B parameters on complex multimodal reasoning benchmarks > <ref:2606.07000#pg3,consistently improves over RLVR and distillation-based baselines>

Meng: The results show PTD-PO maintains higher policy entropy and improves recovery from failed rollouts, which is important for robustness >

Tom: So, the main implication is that we can get dense reasoning guidance while preserving exploration during RLVR post-training >

Jane: It suggests that this method works well across different model scales and modalities because the hints are constructed to be grounded in spatial and textual steps >

The paper's summary: Tom: Now, let's talk about the specific improvements they propose in this paper. They highlight a few things that make PTD-PO different from what came before.

Jane: First is the Privileged Tutoring for RLVR part itself, which is identifying that sparse outcome rewards and solution-revealing self-distillation can limit exploration in multimodal RLVR >

Lu: Then they propose PTD-PO, which routes dense supervision to failed trajectories by conditioning a frozen reference model on spatial and textual hints, while keeping the student under the original answer-free context >

Tom: And they mention the Asymmetric Context Top-K Jensen-Shannon divergence objective as a key technical improvement for stabilizing distillation under distribution shift >

Meng: That divergence objective is specifically designed to stabilize alignment when the teacher and student distributions are conditioned on different contexts, and it reduces the memory cost from O(BT V) to O(BTK) >

Jane: And they also mention that they use a selective PTD objective where they only apply distillation to failed trajectories defined by a criterion where the group success count is small >

Lu: That selective application prevents overly regularizing successful trajectories, which is an important control mechanism for the training process >

Tom: So these improvements boil down to using structured, answer-free hints that are routed only when things go wrong and stabilizing the distillation objective with this specialized divergence metric >

The paper's improvements: Jane: We’re wrapping up this look at "Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimization." Basically, they show that PTD-PO provides dense guidance without exposing the answer to the student policy >

Tom: It's a method designed to complement sparse reward RLVR with dense token-level guidance by using hints that focus on spatial and reasoning steps >

Lu: So it works across different model scales and modalities, showing that this approach can preserve exploration during RLVR post-training >

Meng: It does reduce the memory overhead of online distillation compared to other methods when you are running things at scale >

Jane: The main thing is that answer-free privileged hints are more suitable for guiding failed multimodal reasoning trajectories while preserving exploration, which is a key finding here >

Tom: This method is compatible with different RLVR optimizers, suggesting it’s decoupled from the specific RLVR optimizer and can be an auxiliary distillation module >

Lu: So, PTD-PO provides dense guidance without exposing the answer to the student policy and that’s a strong point >

Jane: It's a solid direction for improving how we use these models when they are learning in complex scenarios, and we’re looking forward to seeing how this helps in practice >

Conclusion: Tom: So, to recap, we're talking about "Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimization," and it’s about how you can give dense guidance to failed rollouts in RLVR without letting the student policy see the actual answer >

Jane: Right. The core idea is that instead of giving them the final solution, you condition a frozen reference model on spatial hints and textual steps to show them *how* to correct their mistakes >

Lu: It’s using those hints to create structured privileged hints offline, which then guide the self-distillation process step by step while keeping the student policy in that original answer-free context >

Meng: The selective part is important too. They only apply this heavy distillation on failed trajectories, so you aren't over-regularizing successful runs >

Lalam: I’m confident this method provides dense token-level supervision when the scalar reward signal is underinformative, which helps recover from those failures >

Tom: It definitely keeps policy entropy higher than what we see in baseline RLVR methods, and that’s good because it means they keep exploring instead of collapsing too fast >

Jane: And they use this Top-K Jensen-Shannon divergence objective to stabilize the alignment between the teacher and student distributions, even when those contexts are different >

Lu: That distribution shift handling is key; it keeps the distillation stable without inducing overly strong gradients that might just push them toward a single deterministic path >

Meng: From a deployment standpoint, if you can get this dense guidance efficiently, it means less compute for the online distillation part and better stability in production >

Tom: So they've shown that these answer-free privileged hints are more useful than just random hints when the task involves spatial reasoning and multimodal understanding >

Jane: They prove that this approach works across different model sizes, from two billion to eight billion parameters, which is encouraging for generalization >

Lu: The main caveat they flag is that it doesn't eliminate all reasoning failures in extremely complex problems where the hint itself just isn't enough to resolve the core difficulty >

Meng: That makes sense. If the hint isn't strong enough, you still run into a failure, even with this dense guidance >

Lalam: I’m sure that limitation points toward constructing even harder training data for these models to really maximize the benefit of this privileged information distillation >

Tom: True. So while PTD-PO is solid for recovering from failures, the next challenge is making those hints even better and more capability-matched >

Jane: It definitely sets a new direction for how we think about providing supervision in RLVR tasks where outcomes are sparse and complex >

Shizhe Xiang, Ke An, Wenlong Yu, Yue Liu, Jian Luan, Pei Fu, Qilong Wang

Tianjin University · Beijing Institute of Technology · Singapore Management University

cs.AI

Submitted: 2026-06-05

Updated: 2026-10-20

Code: https://github.com/XszNeverSleep/PTD-PO

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: The gist The proposed PTD-PO framework provides dense token-level guidance for failed rollouts by conditioning a self-teacher on non-answer-revealing spatial and textual hints, while keeping the

Key concepts

Privileged Tutoring for RLVR
This involves using a frozen reference model as a tutor. It conditions this tutor on spatial attention guidance and intermediate reasoning steps to generate dense, token-level corrections for failed rollouts. This provides step-wise supervision without revealing the final answer to the student policy.
PTD-PO Framework
This is the core method that constructs structured privileged hints from spatial attention and textual reasoning. It applies this guidance only to failed trajectories, using a specific token selection criterion based on group difficulty, ensuring dense supervision for mistakes.
Asymmetric Context Top-K JSD
This objective stabilizes distillation by measuring the divergence between the teacher's hint-augmented distribution and the student's distribution. It uses a Top-K support and tail bucket to reduce memory overhead while focusing alignment on informative token probabilities, preventing unstable gradients.
Failure-Routed Distillation
This mechanism selects specific token positions for PTD supervision based on group difficulty. It ensures that dense guidance is applied precisely to incorrect responses when the activation threshold is set appropriately, avoiding unnecessary regularization on successful trajectories.

Terminology

Summary

The gist The proposed PTD-PO framework provides dense token-level guidance for failed rollouts by conditioning a self-teacher on non-answer-revealing spatial and textual hints, while keeping the student under the original answer-free context.

Motivation and Problem

Reinforcement Learning with Verifiable Rewards (RLVR) often suffers from sparse verifiable rewards that provide little token-level supervision for failed rollouts, leading to inefficient exploration in complex multimodal reasoning tasks 5, the sparse nature of verifiable rewards provides little token-level supervision for failed rollouts, often leading to inefficient exploration in complex multimodal reasoning tasks 1>. Online Policy Distillation (OPD) requires substantial computational overhead and may introduce tokenizer or vocabulary mismatch when using external teachers 2, Online Policy Distillation (OPD) provides such guidance by aligning student rollouts with a teacher policy during training 4>. Existing self-distillation methods often use ground-truth answers or complete solutions as conditioning signals, which may induce answer-level shortcuts, resulting in overly deterministic trajectories, sharpened token distributions, accelerated entropy collapse, and reduced exploration of alternative reasoning paths 5>.

PTD-PO Framework

The PTD-PO framework addresses these limitations by using answer-free hints as tutoring signals for self-distillation 3, PTD-PO uses answer-free hints as tutoring signals for selfdistillation 5>. The core mechanism involves constructing structured privileged hints from spatial attention guidance and intermediate textual reasoning steps 5, PTD-PO constructs structured privileged hints from spatial attention guidance and intermediate textual reasoning steps 5>. These hints are constructed offline by prompting a strong model to reverse engineer an answer-consistent but non-revealing guidance signal from (x, a⋆) 17, The external model is used only as a hint generator before PTD training, and is not involved in rollout, distillation, or evaluation 17>.

Failure-Routed Distillation

PTD-PO routes dense supervision to incorrect responses by selecting token positions based on group-level difficulty 8, we route PTD supervision to the incorrect responses in the sampled group 5>. The set of token positions selected for PTD is defined by F = (i, t) r¯(x) < τptd, ri = 0, mi,t = 1 8. This ensures that PTD is applied to all failed trajectories when the activation threshold τptd is set to 1.0 8>. The final training objective combines the GRPO loss with the PTD regularizer: LPTD-PO = −JGRPO(θ) + λPTDLPTD 12, The final training objective combines the GRPO loss with the PTD regularizer 12>.

Asymmetric-Context Top-K JSD

To stabilize distillation under distribution shift between guided and unguided contexts, a Top-K Jensen-Shannon divergence objective with tail compensation is introduced 5, we introduce a Top-K Jensen-Shannon divergence objective that focuses alignment on informative token probabilities while reducing memory overhead 5>. This objective computes the divergence between the hint-augmented teacher distribution qh and the student distribution pi,t using compressed distributions derived from a Top-K support and a tail bucket to reduce memory cost from O(BT V) to O(BTK) 14, The resulting complexity becomes O(BTK), K ≪ V, up to a small constant factor from the union support and the tail bucket 24>.

Empirical Effectiveness

Experiments on LVLMs ranging from 2B to 8B parameters show that PTD-PO consistently outperforms RLVR and distillation baselines, mitigates entropy collapse, and improves complex multimodal reasoning performance 3, PTD-PO consistently outperforms RLVR and distillation baselines 3>. PTD-PO maintains higher policy entropy, improves recovery from failed rollouts, and reduces the memory overhead of online distillation 3>. Ablation studies confirm that removing structured hint design yields smaller gains, indicating that effectiveness relies on carefully constructed privileged hints 8, the effectiveness of PTD-PO relies on carefully constructed privileged hints that provide grounded, non-spoiling reasoning guidance rather than arbitrary answer-free hints 8>.

Conclusion and Generalizability

PTD-PO demonstrates that privileged tutoring distillation provides effective dense guidance beyond sparse outcome-level rewards 3, PTD-PO provides dense guidance without exposing the answer to the student policy 3>. The method is compatible with different RLVR optimizers, suggesting its general applicability across various policy optimization objectives 8, PTD is designed as an auxiliary distillation module and is therefore decoupled from the specific RLVR optimizer 9>.

Limitations and Future Work

A limitation noted is that PTD-PO does not fully eliminate reasoning failures in highly compositional multimodal problems, suggesting that when the hint is insufficient to resolve the core difficulty, the model may over-rely on partial guidance 27. Future work suggests constructing harder and more capability-matched RLVR training data for stronger LVLMs to enlarge the benefit of privileged-information distillation 28, Constructing harder and more capability-matched multimodal RLVR training data may further enlarge the benefit of privileged-information distillation, especially for stronger models 28>.

References

REFERENCES

Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from selfgenerated mistakes. In The twelfth international conference on learning representations, 2024.

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861.

Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631.

Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang et al. Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37:27056–27087, 2024.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong et al. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems, 36:49250–49267, 2023.

Yihe Deng, Hritik Bansal, Fan Yin et al. Openvlthinker: An early exploration to complex vision-language reasoning via iterative self-improvement. arXiv e-prints, pp. arXiv–2503, 2025.

Ken Ding. Hdpo: Hybrid distillation policy optimization via privileged self-distillation. arXiv preprint arXiv:2603.23871.

Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Yuanheng Zhu et al. Revisiting on-policy distillation: Empirical failure modes and simple fixes. arXiv preprint arXiv:2603.25562.

Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models.

Improvements for AI systems

  1. Privileged Tutoring for RLVR: The system can provide dense guidance for failed rollouts by conditioning a frozen reference model on spatial and reasoning hints to provide token-level guidance on how the failed reasoning path should be corrected.

  2. PTD-PO Framework: The system will generate structured privileged hints from spatial attention guidance and intermediate textual reasoning steps to produce step-wise token-distribution supervision while the student policy remains under the original answer-free context.

  3. Asymmetric Context Top-K JSD: The system will use a Top-K Jensen-Shannon divergence objective with tail compensation to stabilize distillation, ensuring that the teacher and student distributions are conditioned on different contexts without inducing overly strong and unstable gradients.

  4. Selective PTD Objective: The system will apply distillation only to failed trajectories, defined by the criterion where the group success count is small, preventing over-regularization on successful trajectories.

  5. Enhanced Reasoning Recovery: The improved AI can recover failed trajectories by injecting a distributional correction when reward-induced targets are under-informative, as PTD complements the scalar reward signal with a dense token-level target generated from the hint-conditioned self-teacher.

  6. Improved Generalization: The system will show that answer-free privileged hints are more suitable for guiding failed multimodal reasoning trajectories while preserving exploration, suggesting better performance across different model scales and modalities.

Abstract

Recent post-training methods, particularly Reinforcement Learning with Verifiable Rewards (RLVR), have significantly enhanced the reasoning ability of Large Vision-Language Models (LVLMs). However, the sparse nature of verifiable rewards provides little token-level supervision for failed rollouts, often leading to inefficient exploration in complex multimodal reasoning tasks. Although policy distillation can offer dense guidance, external teacher based methods introduce substantial computational overhead, while answer conditioned tuning methods may expose answer-level information and induce shortcut-like generation behavior. To address these limitations, we propose PTD-PO, a Privileged Tutoring Distillation Policy Optimization framework for RLVR that provides dense guidance without exposing the answer to the student policy. Specifically, PTD-PO constructs structured privileged hints from spatial attention guidance and intermediate textual reasoning steps, and uses them through in-context learning to produce step-wise token-distribution supervision. The student is still optimized under the original answer-free context, and its failed rollouts are aligned with the hint-augmented reference model at the token-distribution level. To further stabilize distillation under the distribution shift between guided and unguided contexts, we introduce a Top-K Jensen-Shannon divergence objective that focuses alignment on informative token probabilities while reducing memory overhead. Experiments on LVLMs ranging from 2B to 8B parameters show that PTD-PO consistently outperforms RLVR and distillation baselines, mitigates entropy collapse, and improves complex multimodal reasoning performance.

Sources

Related papers