State-OPD: State-Grounded On-Policy Distillation for Multi-Turn Agents

arXiv:2610.13238 · cs.LG · Submitted 2026-10-09 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "State-OPD: State-Grounded On-Policy Distillation for Multi-Turn Agents".

Jane: The gist On-Policy Distillation (OPD) has emerged as a powerful paradigm for transferring reasoning capabilities from frontier models to smaller models,

Tom: First, who's behind it and why it matters.

Paper summary: Lu: The authors of "State-OPD: State-Grounded On-Policy Distillation for Multi-Turn Agents" conclude by summarizing their approach of grounding teacher targets in executed actions and observations and resampling complete responses at high-entropy states while retaining full history for student generation.

Tom: So, to put it simply, they ground the teacher’s guidance on what actually happened—the actions and observations—and they focus their sampling effort on the most complex decision points where the agent has many different ways to proceed.

Jane: And they keep letting the student keep generating its full history so it doesn't lose that context entirely, even though they are heavily conditioning the teacher on just that specific interaction state.

Meng: It seems like a practical method because it’s not trying to reinvent the entire reasoning process; it’s just refining how we use existing distillation techniques for multi-turn agents.

Lu: They demonstrate that this approach outperforms other methods across three different benchmarks, and it even surpasses the teacher model on ALFWorld Seen for Qwen3-4B students <ref:2610.13238#pg1>.

Tom: The big picture is that separating the realized interaction from what the student thinks about it gives us a common foundation for both grounding supervision and exploring alternative decisions at the same state.

Jane: So, if you’re listening to this show, this paper suggests that when teaching an AI to handle multi-turn tasks, you should focus on anchoring its learning in the physical actions it actually takes rather than just its internal thoughts.

Lu: That is the core idea driving State-OPD: grounding teacher targets in executed actions and observations and reallocating the sampling budget to states with high response entropy while retaining full history for student generation.

Conclusion: Tom: So we're wrapping up with State-OPD, and the title itself—"State-Grounded On-Policy Distillation for Multi-Turn Agents"—it sounds a bit heavy, right?

Jane: It does sound like it is, Tom. It’s basically taking something we already know, on policy distillation for smaller models, and trying to make it work reliably in those long agent conversations.

Lu: Exactly. The paper is about grounding the teacher's supervision not just in what the student has done so far, but specifically in a defined interaction state.

Meng: And they do this by reallocating their sampling budget to focus on states where the AI is making many different choices, those high-entropy states.

Tom: So what does this actually mean for the people listening? It’s about making sure when we teach these smaller models to reason over several steps, we don't just hand them a massive wall of text and hope they get it right.

Jane: Right. The implication is that instead of treating every turn as a new, isolated thing, you condition the teacher on the context of where the agent *is* in its task—the state—which seems to be much more stable than focusing only on the last few words spoken.

Lu: They show that removing prior reasoning from that teacher context actually helps it continue better when interactions get longer.

Tom: That’s a clean finding, Lu. It means the student's internal thoughts aren't always helping the teacher guide the next move; sometimes they just add noise to the signal.

Jane: And they prove that by focusing on those specific interaction states and resampling complete responses there, you get better action diversity than if you just sampled actions while holding onto a fixed reasoning trace.

Meng: From a deployment standpoint, that sounds like it could lead to much more robust agents because they aren't repeating the same mistakes in the same way across different turns.

Tom: So we’ve seen how they ground the supervision and where they focus their sampling. What’s next is seeing how this performs on real-world tasks compared to other methods.

Zhentao Zhang, Yanzhao Zheng, Yanlin Liu, Yuanqiang Yu, Tianze Xu, Chao Ma, Jihuai Zhu, Lelun Wu, Baohua Dong

Alibaba Group University of Science and Technology of China University of Science and Technology of China

cs.LG

Submitted: 2026-10-09

Updated: 2026-10-09

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: The gist On-Policy Distillation (OPD) has emerged as a powerful paradigm for transferring reasoning capabilities from frontier models to smaller models, while extending its success to multi-turn

Key concepts

Interaction State: This is a snapshot of the agent's situation at a specific turn, defined by the sequence of executed actions and observations, excluding past internal reasoning. The paper proves this state captures all necessary environmental information from the full history.
Prior Reasoning Distortion: The student's reasoning that occurred before the current step can mislead teacher supervision because it adds no new environmental context beyond what is already captured in the interaction state. Removing this prior reasoning improves the teacher's ability to guide subsequent steps.
State-Grounded Supervision: This framework conditions the teacher on a specific interaction state while allowing the student to generate responses from its entire history. It grounds supervision in observed actions and environmental feedback, ensuring the teacher focuses on relevant context.

Terminology

Distillation — 1× in this paper · explained in 2 episodes
On-Policy Distillation — 1× in this paper · explained in 3 episodes

Summary

The gist On-Policy Distillation (OPD) has emerged as a powerful paradigm for transferring reasoning capabilities from frontier models to smaller models, while extending its success to multi-turn agentic environments remains challenging State-OPD introduces a framework that grounds teacher supervision in interaction states and reallocates the sampling budget to generate multiple complete reasoning–action responses at states with high response entropy, while retaining full-history student generation.

How it works

The core idea is to distinguish between the objective task and executed action–observation record from the internal semantic trajectory of student reasoning The paper identifies two roles of student reasoning in multi-turn OPD: prior reasoning distorts teacher supervision, while current reasoning shapes the action alternatives sampled at the same state. Specifically, at matched interaction states, removing prior student reasoning improves teacher continuation success and changes the teacher’s token-level targets. Meanwhile, resampling complete reasoning–action responses yields greater action diversity at the same state than resampling actions under a fixed reasoning trace.

State-Grounded Supervision

State-OPD organizes teacher supervision and student response sampling around interaction states Stategrounded supervision conditions the teacher on the interaction state, while the student continues to generate responses from its full history. The loss function for State-OPD is defined as LState-OPD(θ) = X (i,t,r)∈R X r k=1 DKL(πθ(· hi,t, r<k)∥ πT (· si,t), where R denotes the indexed response pool for an update and si,t is the interaction state. State grounding therefore removes the preceding student reasoning z<t only from the teacher’s context; the student’s on-policy generation context and the current autoregressive response prefix remain unchanged.

State-Centered Response Allocation

To bring more reasoning-induced actions under supervision within a fixed nominal turn-response budget, State-OPD concentrates complete-response samples at selected interaction states. The selection criteria involve computing the student’s full-vocabulary token entropy averaged over its realized tokens, Hi,t = 1/Li,t X Li,t k=1 H[πθ(· hi,t, ri,t,<k)]. For a root trajectory with Ti turns and reference multiplicity M = 4, the number of additional responses sampled at each selected state is n = β(M − 1), where Pi = Ti/β. This reallocates the budget toward more responses at fewer states.

Key Findings and Results

Comprehensive evaluations across ALFWorld, ScienceWorld and WebShop demonstrate that State-OPD achieves the highest macro-average success rate among the evaluated methods for both Qwen3-1.7B and Qwen3-4B students. Notably, for Qwen3-4B, State-OPD raises macro-average success from 26.07% with the strongest baseline to 31.74% and surpasses its Qwen3-32B teacher on ALFWorld Seen. Training and behavioral analyses on ALFWorld further show sustained gains with fewer supervised turn responses and a substantial reduction in persistent failed-action loops. State-OPD revisits interaction evidence and breaks failed-action loops.

Ablation Studies

A two-factor ablation on ALFWorld with a Qwen3-4B student and Qwen3-32B teacher shows that state grounding conditions the teacher target on st, response allocation brings more reasoning-induced actions at the same state into supervision, and their combination achieves the highest success on ALFWorld. Specifically, State grounding raises average SR from 32.42% to 41.73% without response allocation and from 35.35% to 44.55% with it. Response allocation raises average SR from 32.42% to 35.35% under full-history teacher conditioning and from 41.73% to 44.55% under state-grounded conditioning.

Conclusion

State-OPD grounds teacher targets in executed actions and observations and resamples complete responses at high-entropy states, while retaining full-history student generation. Across three benchmarks, State-OPD achieves the highest macro-average success among trained methods at both student scales; its Qwen3-4B student also surpasses the Qwen3-32B teacher on ALFWorld Seen. On ALFWorld, it sustains higher success with fewer supervised turn responses and sharply reduces persistent failedaction loops. Separating the realized interaction from the student’s reasoning about it provides a common foundation for grounding teacher supervision and exploring alternative decisions at the same state.

--- Page 9 ---

Interaction State Sufficiency

We define the interaction state at turn t as st = (g, o0, a0, o1,..., at−1, ot), excluding z<t. Proposition 1 proves that p(ξt ht) = p(ξt st), which shows that st preserves all information in ht about the latent environment state ξt. This suggests that histories with different reasoning can share the same st when their executed actions and returned observations are identical.

Removing Inherited Reasoning Improves Teacher Guidance

At a fixed interaction state, the teacher already has the complete record of executed actions and environmental feedback. By Proposition 1, prior student reasoning adds no environment-state information beyond st, yet vanilla OPD includes it in the teacher context used to form the supervision target. Analysis shows that Ts achieves higher continuation success than Th for both student models in every anchor range. The growing ∆T (d) indicates that inherited student reasoning increasingly weakens the teacher’s ability to guide the task as interactions deepen.

A Single Reasoning Path Limits Action Diversity

In each student response rt = (zt, at), the sampled reasoning path precedes and shapes the sampled action. Testing complete-response resampling reveals that 88.5% of fixed-reasoning probes yield just one parsed action, compared to 44.3% of complete-response probes. This indicates that varying current reasoning thus exposes action alternatives that sampling only the action suffix misses at the same interaction state.

Improvements for AI systems

Bold header: Grounding Teacher Supervision in Interaction States

State-OPD "grounds teacher supervision in interaction states and reallocates the sampling budget to generate multiple complete reasoning–action responses at states with high response entropy, while retaining full-history student generation. This allows the teacher to condition on the interaction state, which conditions the teacher on the interaction state, improving supervision quality by removing prior student reasoning that distorts supervision."

Bold header: Enhancing Action Diversity via State-Centered Response Allocation

The framework employs a strategy where it applies a concentration factor β to reallocate this budget toward more responses at fewer states, specifically targeting states with high response entropy to increase action diversity, as shown by the finding that entropy-ranked state selection recovers more distinct actions than random selection at every tested β under matched budgets.

Bold header: Improving Decision Revision and Reducing Failed Action Repetition

State-OPD achieves a substantial reduction in persistent failed-action repetition because it allows the student to revisit interaction evidence and breaks failed-action loops, as demonstrated by the observation that State-OPD is substantially less likely to persist with an action that the environment has already rejected.

Bold header: Sustaining Performance Over Training Time

The system demonstrates sustained performance gains, showing that State-OPD’s ALFWorld advantage persists through training, reaching higher success and fewer interaction rounds at later checkpoints compared to Vanilla OPD, as shown by the finding that the growing ∆T (d) indicates that inherited student reasoning increasingly weakens the teacher’s ability to guide the task as interactions deepen.

Bold header: Robustness Against Context Contamination

By defining the interaction state st as the task instruction and the sequence of executed actions and returned observations, excluding prior student reasoning, State-OPD isolates reasoning's effect, allowing it to effectively handle shared context compression that degrades supervision in traditional multi-turn OPD.

Abstract

On-Policy Distillation (OPD) has emerged as a powerful paradigm for transferring reasoning capabilities from frontier models to smaller models, while extending its success to multi-turn agentic environments remains challenging. We study agentic OPD through interaction states, distinguishing the objective task and executed action-observation record from the internal semantic trajectory of student reasoning. Through this lens, we identify two roles of student reasoning in multi-turn OPD: prior reasoning distorts teacher supervision, while current reasoning shapes the action alternatives sampled at the same state. Specifically, at matched interaction states, removing prior student reasoning improves teacher continuation success and changes the teacher's token-level targets, with larger continuation gains at later turns. Meanwhile, resampling complete reasoning-action responses yields greater action diversity at the same state than resampling actions under a fixed reasoning trace. We introduce State-Grounded On-Policy Distillation, a framework that grounds teacher supervision in interaction states and reallocates the sampling budget to generate multiple complete reasoning-action responses at states with high response entropy, while retaining full-history student generation. Comprehensive evaluations across ALFWorld, ScienceWorld and WebShop demonstrate that State-OPD achieves the highest macro-average success rate among the evaluated methods for both Qwen3-1.7B and Qwen3-4B students. Notably, for Qwen3-4B, State-OPD raises macro-average success from 26.07% with the strongest baseline to 31.74% and surpasses its Qwen3-32B teacher on ALFWorld Seen. Further analyses reveal that State-OPD achieves higher task success with fewer supervised turn responses and substantially reduces persistent failed-action repetition; qualitative traces illustrate decision revision based on interaction feedback.