Multi-Teacher On-Policy Distillation for Flow-Based Vision-Language-Action Models

arXiv:2610.21522 · cs.RO · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Multi-Teacher On-Policy Distillation for Flow-Based Vision-Language-Action Models".

Dev: The gist:

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: To recap, the main thesis of this paper is that specialized vision language action policies are good at specific tasks, but merging those skills into one shared policy is hard. They introduce a two-level on policy distillation framework called Velocity Loss OPD to solve the distribution mismatch when you try to deploy these models in the real world.

Dev: They tackle this by creating a system where frozen domain specialists jointly supervise one student based on student-generated environment interactions and action flow trajectories. This allows for knowledge transfer even when the student is exploring new areas.

Taro: The framework works by having each frozen specialist supply local velocity targets in its specific domain, which are then used to guide the student's learning process.

Rosa: They define two levels of supervision: the environment level on policy component and the action-generation level on policy component. The environment level involves student collection, domain-teacher labeling, and student updates to keep both supervision distributions aligned with the evolving policy.

Dev: And at the action generation level, they query the domain teacher at specific states along the student’s generation trajectory to provide a local velocity target for that particular flow path.

Taro: The Velocity Loss is defined as Lvel,r(θ) = X K k=one wk Ec∼d πθ¯r k, τ∼ρk x∼qθ¯ r,τ (·c) <ref:2610.21522#pg1>. It’s essentially a way to measure the velocity error along the student's flow path.

Rosa: The training protocol starts from a joint supervised fine-tuning initialization and then moves through rounds where each domain teacher provides supervision based on the student's current experience.

Dev: After each round, the student collects new environment experience and generates fresh action-flow trajectories, which refreshes both the supervision distributions for the next update.

Taro: The paper shows that this approach yields absolute success-rate gains of twenty-two point five five percent on LIBERO-forty thirty-three point four percent on CALVIN five instruction sequences, and twenty-two point six percent on Meta-World MT50 from a joint SFT initialization <ref:2610.21522#pg1,yields absolute success-rate gains of 22.55% on LIBERO-40, 33>.

Rosa: The results compared it to other distillation methods, showing that Velocity Loss OPD achieves absolute success-rate gains of six point four to nine point five percent on LIBERO-forty nine point seven to eighteen point three percent on CALVIN, and three point two to four point two percent on Meta-World MT50 from Action Loss OPD and Teacher-rollout SFT respectively.

Dev: What's important is that the Velocity Loss OPD outperforms Action Loss OPD by nine point five zero points on LIBERO, while it also beats Teacher-rollout SFT by six point four zero points on LIBERO and three point two zero points on Meta-World MT50, plus a quarter of an instruction in CALVIN average length <ref:2610.21522#pg3>.

Taro: The analysis by sequence length further shows larger success-rate gains on longer CALVIN sequences when using this two-level supervision method.

Rosa: So, the gist is that this framework consolidates domain specialists into one student across three domains using this two-level on policy supervision strategy. This sets up the next part of our discussion about what these results actually mean for us in practice.

Conclusion: Dev: So looking at the paper "Multi-Teacher On-Policy Distillation for Flow-Based Vision-Language-Action Models," we see that they proposed this two-level on policy distillation framework as a way to consolidate domain skills into one student.

Rosa: It seems the authors are focused on showing that by supervising both environment and action generation states, you get better alignment with the evolving policy.

Dev: They proved it works across LIBERO, MetaWorld, and CALVIN, achieving specific success rate gains like twenty-two point five five percent on LIBERO-forty <ref:2610.21522#pg1,22.55% on LIBERO-40>.

Taro: The implication is that for anyone working on complex VLA models, you can start with this kind of shared student initialized from a joint SFT checkpoint and expect some measurable improvement across domains.

Rosa: It means we don't have to train three separate specialists if we can find a way to distill their knowledge into one central model effectively.

Dev: They also highlighted that the method outperforms other distillation techniques on the metrics they tested, which points toward it being a more robust way to handle these distribution shifts in deployment.

Taro: The main limitation they mentioned is that because of how they set up the two sampling levels, they end up reusing the acting policy's integration path for both environment execution and flow-state supervision.

Rosa: That reused path introduces sequential solver dependencies, meaning you might run into issues with timing or latency when trying to scale this up.

Dev: So while it’s a solid method for consolidating these skills, it shows the consolidation is possible, but we still have to figure out how to make that interaction smoother and more efficient at scale.

Lichao Ni, Yuhan Liu, Pengfei Li, Guanghuan Fang

Huawei Technologies Co. Ltd. · Tsinghua University

cs.RO

Submitted: 2026-10-15

Updated: 2026-10-15

Comments: 13 pages, 6 figures, 4 tables. Includes supplementary material

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 88/100

The gist: The gist: A two-level on-policy distillation framework couples student-driven environment interaction with teacher velocity supervision along student-generated action flow trajectories to consolidate

Key concepts

Two-Level On-Policy Distillation: This framework uses two distinct levels of supervision: one focuses on the student's interaction with the environment (what happens in the world) and another focuses on supervising the student's generated action flow trajectories (how actions are being planned). This dual approach aligns both types of knowledge from domain specialists.
Environment Level Supervision: This level involves the student collecting new environment experience, which is then labeled by a domain teacher. The goal is to ensure that the student's experience distribution matches what the specialist knows about how to interact with specific environments, keeping both distributions aligned during updates.
Action-Generation Level Supervision: This level involves querying a frozen domain teacher at the student's current state to receive a local velocity target for its action generation trajectory. This provides guidance on the intermediate steps of action planning, helping the student learn better flow dynamics specific to that domain.

Terminology

Distillation — 1× in this paper · explained in 2 episodes
On-Policy Distillation — 1× in this paper · explained in 3 episodes
Policy distillation — 1× in this paper · explained in 2 episodes

Summary

The gist: A two-level on-policy distillation framework couples student-driven environment interaction with teacher velocity supervision along student-generated action flow trajectories to consolidate domain-specialized flow-based vision-language-action models into a single shared student, achieving success rate gains across LIBERO, MetaWorld, and CALVIN.

Multi-Teacher Consolidation Strategy

The framework proposes a two-level on-policy distillation framework that couples student-driven environment interaction with teacher velocity supervision along student-generated action flow trajectories The framework couples student-driven environment interaction with teacher velocity supervision along student-generated action flow trajectories Each frozen specialist supplies local velocity targets in its domain. The framework couples student-driven environment interaction with teacher velocity supervision along student-generated action flow trajectories

Two Levels of Supervision

The two levels of supervision are defined as the environment level on-policy component and the action-generation level on-policy component. The environment level on-policy component involves student collection, domain-teacher labeling, and student updates to keep both supervision distributions aligned with the evolving policy The action-generation level on-policy component involves querying the domain teacher at the student’s state x ∼ qθ¯r,τ (· c) to provide a local velocity target on the student’s generation trajectory The Velocity Loss is defined as Lvel,r(θ) = X K k=1 wk Ec∼d πθ¯r k, τ∼ρk x∼qθ¯r,τ (·c)

Training Protocol and Updates

Starting from a common demonstration-SFT initialization πθ0, the goal is to improve the shared student’s performance across domains and recover the capabilities of the specialists. Each round collects the same number of episodes per domain and accumulates weighted domainloss gradients before one optimizer update to the student The updated student then becomes the collecting snapshot for the next round, refreshing both supervision distributions

Performance Gains and Comparison

Velocity Loss OPD yields absolute success-rate gains of 22.55% on LIBERO-40, 33.4% on CALVIN fiveinstruction sequences, and 22.6% on Meta-World MT50. Compared with Action Loss OPD and Teacher-rollout SFT, it achieves absolute success-rate gains of 6.4–9.5% on LIBERO-40, 9.7–18.3% on CALVIN, and 3.2–4.2% on Meta-World MT50 Velocity Loss OPD exceeds Action Loss OPD by 9.50 points on LIBERO, 4.20 points on Meta-World, and 0.618 instructions in CALVIN average length

Sensitivity to Supervision Design

The comparison between supervision objectives shows that Velocity Loss OPD exceeds Teacher-rollout SFT by 6.40 points on LIBERO, 3.20 points on Meta-World, and 0.288 instructions in CALVIN average length The gains demonstrate the advantage of joint student-induced supervision over teacher-induced supervision Furthermore, the main loss weights outperform equal weighting on every aggregate domain metric in the comparison

Conclusion

Our two-level on-policy framework integrates domain specialists by supervising studentinduced environment and action-generation states. It outperforms the SFT initialization, Action Loss OPD, and Teacher-rollout SFT on all aggregate metrics across LIBERO, MetaWorld, and CALVIN Velocity Loss OPD outperformance Action Loss OPD at every post-update evaluation On CALVIN, Velocity Loss OPD achieves larger success-rate gains over Teacher-rollout SFT as instructionsequence length increases. The framework consolidates domain specialists into one student The two-level on-policy framework integrates domain specialists by supervising studentinduced environment and action-generation states

Supplementary Details

The shared π0.5 student is initialized from the joint SFT checkpoint at step 14,000. All three distillation methods use the same domain teacher configurations, identified as follows • LIBERO: pi05 libero; • Meta-World: RLinf-Pi05-MetaWorld-RL-FlowSDE; • CALVIN: RLinf-Pi05-CALVIN-ABC-D-RL-FlowSDE The domain weights used are LIBERO / MetaWorld / CALVIN: 0.25/0.25/0.50. Collection n = 64 episodes per domain; 192 total per round; one optimizer update The action interface settings are (H, m, N) = (5, 5, 5) for Meta-World and CALVIN where H is the chunk horizon, m the executed prefix, and N the solver-step count. The training updates the action expert and rank-64 language LoRA while freezing the vision backbone The optimizer used is AdamW with betas (0.9, 0.95), epsilon 10−8, weight decay 10−10 The collection order is LIBERO → Meta-World → CALVIN

Evaluation Scope

Each method is evaluated from a single training run under common optimization settings, with matched update and episode budgets. Meta-World rankings vary during training LIBERO and Meta-World use the collection initial-state pools for evaluation CALVIN uses A/B/C student collection and scene-D evaluation, with prior teacher exposure to D unspecified The comparison of two mixed-source alternatives measures the joint effect of changing both sampling distributions without isolating each level’s contribution

Domain Weighting Analysis

Equal weighting is lower by 7.65 percentage points on LIBERO, 3.60 points on Meta-World, and 0.225 instructions on CALVIN Within LIBERO, it improves Spatial and Goal to 96.2% and 69.4%, while Object falls from 98.4% to 89.2% and LIBERO-10 from 46.4% to 17.4% The main weights yield higher aggregate scores, while equal weighting improves two LIBERO suites The main weights improve LIBERO-10 by 29.0 percentage points over equal weighting despite assigning LIBERO a smaller relative weight

Discussion on Longer Tasks

Two-level on-policy supervision delivers pronounced gains on LIBERO-10 and CALVIN sequence completion Relative to Teacher-rollout SFT, which uses the same velocity objective, the LIBERO10 gain is 13.8 percentage points, compared with 1.6–5.8 points on the other three LIBERO suites This pattern is consistent with compounding distribution shift in longer rollouts Two-level OPD places teacher guidance on both the student’s visited environment states and its own intermediate flow states providing supervision along these evolving trajectories The consolidated student improves all three benchmarks but still falls short of the corresponding specialists in aggregate performance A promising direction is to harmonize action semantics across domains by using aligned action representations within each domain This could facilitate parameter sharing and narrow the remaining teacher gap

Limitations and Future Directions

The cost of decoupling the two sampling levels means that Velocity Loss OPD and Teacher-rollout SFT both reuse the acting policy’s integration path for environment execution and flow-state supervision Generating this additional path introduces sequential solver dependencies whereas queries at already available path states can be batched This comparison measures the joint effect of changing both sampling distributions without isolating each level’s contribution A unified action interface could assign consistent semantic slots and gripper conventions while mapping motion commands into compatible coordinate frames and scales The framework still falls short of the corresponding specialists in aggregate performance The consolidated student combines improvements across all three domains

References

[1] Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations, 2024

[2] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones. π0: A vision-language-action flow model for general robot control.

Improvements for AI systems

Robust cross-domain capability consolidation through dual supervision levels: The framework couples student-driven environment interaction with teacher velocity supervision along student-generated action flow trajectories. This allows a single shared student to accommodate distinct observation and native action interfaces while integrating specialist supervision through the same learned parameters, leading to performance gains of 22.55% on LIBERO-40, 33.4% on CALVIN fiveinstruction sequences, and 22.6% on Meta-World MT50.
Enhanced long-horizon task completion: By utilizing the two-level supervision, the system demonstrates that Velocity Loss OPD achieves larger success-rate gains over Teacher-rollout SFT as instruction sequence length increases, explaining why gains are significant for longer CALVIN sequences by providing guidance on both environment and action-generation states.
Superior performance in complex benchmarks: The proposed method outperforms the SFT initialization, Action Loss OPD, and Teacher-rollout SFT on all aggregate metrics across LIBERO, Meta-World, and CALVIN, indicating a more effective consolidation of domain-specialized skills into one policy.