What Should Robot Policies Learn to Predict? Rethinking Supervision Targets in Vision-Language-Action Learning

arXiv:2610.21915 · cs.RO · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "What Should Robot Policies Learn to Predict? Rethinking Supervision Targets in Vision-Language-Action Learning".

Dev: The gist: suitable LA and FV supervision can outperform action-only training under the same training budget, even when direct action prediction receives no more than half of the samples,

Rosa: First, who's behind it and why it matters.

Paper summary: Dev: So looking at the full picture of this paper, What Should Robot Policies Learn to Predict? Rethinking Supervision Targets in Vision-Language-Action Learning. It really boils down to saying that for vision language action policies, we shouldn't just rely on training them to predict the next action every single time.

Rosa: The authors show that by using a unified framework with sample-level routing, you can explicitly control how much training goes to direct action prediction versus auxiliary tasks like latent actions or future vision.

Taro: What this means for us is that if we design our supervision targets carefully, we don't have to dump all our training budget into just one thing. We can tailor the learning process to what the robot needs to learn most effectively.

Dev: The findings show that in some cases, like on LIBERO and all four real-robot tasks, these alternative supervision methods beat training purely focused on action prediction when the budget is fixed at half or less for direct action prediction.

Rosa: And the paper points out that while future vision often leads the pack on average across those tasks, there are specific targets, like LAPA-DINOv3 on CALVIN, that stay competitive even if they aren't the best overall performer.

Taro: So the implication is that we need to think about what kind of predictive signals actually help a robot perform its job in the real world and then structure our training to prioritize those signals.

Dev: It shifts the focus from just making sure we get an action prediction loss down, to designing a system where we choose between different types of predictions—relative motion versus absolute position—and allocate the samples accordingly.

Conclusion: Rosa: So we're wrapping up this look at 'What Should Robot Policies Learn to Predict? Rethinking Supervision Targets in Vision-Language-Action Learning'. Basically, we're talking about how training a robot to do something involves picking what kind of prediction signal you supervise it on.

Dev: Right, so the paper looks at why we usually just train the robot to predict the next action directly and whether using other things like latent actions or future vision helps when you have limited time.

Taro: It's about this idea that maybe focusing on predicting something else—like relative motion versus absolute position—actually gives the policy better skills for the job, even if it doesn't predict the action itself every time.

Rosa: The authors show that using a unified framework with sample routing lets you explicitly choose whether to supervise action prediction or one of these other targets like LAPA or DINOv3.

Dev: And the main point is that under a fixed budget, you can actually get better results by giving some samples to these auxiliary goals instead of just sticking with action prediction all the time.

Taro: They found that for real-world tasks, future vision targets tend to perform better than latent action targets on average across many different robot setups.

Rosa: But there are still trade-offs, like how DINO-based targets focus on relative motion while future vision focuses on absolute position and task identity.

Dev: That’s the nuance, right? It means the choice of supervision target isn't just some arbitrary setting; it actually shapes what the robot learns about the world.

Taro: So, we might need to stop defaulting to action-only training and start thinking about which specific predictive targets make sense for a given task or environment.

Rosa: It really shifts the design choice from just 'how do I get an action prediction loss down?' to 'what kind of prediction should this robot be good at making?'

Haitao Shen, Hanyu Liu, Yihan Lin, Huiping Liu, Junbo Cui, Daoheng Li, Yang Li, Jing Zhang, Wenxin Li

Renmin University of China · Peking University

cs.RO

Submitted: 2026-10-15

Updated: 2026-10-15

Comments: 26 pages, 8 figures. Haitao Shen and Hanyu Liu contributed equally

Code: https://github.com/meituan-longcat/LARYBench

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 83/100

The gist: The gist: suitable LA and FV supervision can outperform action-only training under the same training budget, even when direct action prediction receives no more than half of the samples, suggesting

Key concepts

Action Prediction: This is the primary way robot policies are usually trained, where the model learns to directly predict what physical actions a robot should take based on its current state and vision. The paper investigates if this single focus is the best approach for learning.
Latent-Action (LA) Targets: These are auxiliary prediction goals where the policy learns to predict an internal, hidden representation of an action. The study uses targets based on LAPA and DINO features, which help the policy understand relative motion between states.
Future-Vision (FV) Targets: These targets train the policy to predict future visual information or flow velocity. Using representations like DINOv3 and V-JEPA 2, these help the robot learn about absolute positions and task identity in the future.

Summary

The gist: suitable LA and FV supervision can outperform action-only training under the same training budget, even when direct action prediction receives no more than half of the samples, suggesting that what robot policies learn to predict and how training is allocated among these targets should be treated as explicit design choices rather than defaulting to action-dominant training.

How it works

The paper introduces a unified VLA framework that couples a two-stream multimodal diffusion transformer (MMDiT) with a sample-level router, which assigns each training sample to either action prediction or an auxiliary objective and thus makes supervision allocation explicit. This framework allows for the controlled comparison of how different supervision targets shape policy learning.

The framework instantiates three LA targets based on LAPA and its DINO-feature variants, and three FV targets based on DINOv3, V-JEPA 2, and Wan2.2 representations. The sample-level training router determines the active loss for each training sample under a fixed budget B=NBeff. This router samples a routing indicator ri ∼ Bernoulli(pA), where ri = 1 denotes the action route and ri = 0 denotes the auxiliary route, with pA being P25, P50, P75, and P100.

The auxiliary expert configurations differ for LA and FV supervision. For LA supervision, the expert predicts zˆt ∼ πLAθ(· QLA, ht), where QLA carries no source-specific features. For FV supervision, the auxiliary expert predicts the flow velocity of the future visual representation: vˆf t+δ = vFVθ ft, f τf t+δ, at, ht, τf.

Experimental Findings and Comparisons

The study evaluates the framework on two simulation benchmarks (LIBERO and CALVIN) and four real-robot manipulation tasks. A main finding is that suitable LA and FV supervision can outperform action-only training under the same training budget, even when direct action prediction receives only half or fewer of the training samples.

Specifically, FV generally outperforms LA across both simulation and real-robot tasks, although the strongest LA target remains competitive on CALVIN. Diagnostic analyses suggest that DINO-based LA targets emphasize relative motion, whereas FV targets emphasize absolute position and task identity.

Representation Characteristics

Diagnostic analyses revealed differences in what LA and FV shape policy learning. DINO-based LA targets emphasize relative motion, whereas FV targets better capture absolute position and task identity, and both can alter policy attention toward task-relevant regions.

Further analysis of frozen source representations showed that both LA and FV carry action-related information, so predicting such targets may provide indirect action supervision on auxiliary-routed samples. LAPA-DINO targets emphasize relative motion, while FV targets emphasize absolute position and task identity.

Conclusion

Robot policies must execute actions, but this does not require most training samples to supervise direct action prediction. In a unified VLA framework with sample-level routing, suitable LA and FV targets can outperform action-only training under the same budget even when direct action prediction receives no more than half of the samples, although the best allocation depends on the target, benchmark, and budget. Between the two families, FV generally performs better, leading on LIBERO and all four real-robot tasks. While LAPA-DINOv3 remains competitive on CALVIN. Diagnostics suggest why both can help: both carry action-related information, with LAPA-DINO targets emphasizing relative motion and FV targets emphasizing absolute position and task identity, and both can redirect policy attention toward task-relevant regions. This findings support designing predictive targets and their supervision allocation jointly, rather than defaulting to action-only training.

Improvements for AI systems

Bold header: Adaptive Supervision Allocation

The improved system will dynamically allocate training samples between direct action prediction and auxiliary objectives (LA or FV) based on explicit design choices, as demonstrated by the sample-level router that assigns each training sample to either action prediction or an auxiliary objective. This allows the policy to learn what it should predict and how training is allocated, rather than defaulting to action-dominant training.

Bold header: Task-Specific Supervision Prioritization

The system can prioritize supervision based on the specific task and benchmark, as suggested by Figure 3 showing that the appropriate allocation therefore depends on both the target and the benchmark. For instance, it can use FV (Future Vision) when performance gains are sought in long-horizon tasks like CALVIN, while leveraging LA (Latent Action) when emphasizing relative motion is beneficial for certain manipulation dynamics.

Bold header: Enhanced Relative Motion Sensitivity

By utilizing DINO-based LA targets, the policy will be explicitly trained to capture finer movement details, as these emphasize relative motion, which is shown to provide superior performance on unseen tasks in terms of positional error metrics compared to other sources.

Bold header: Robust Absolute Position and Task Identity Prediction

The system can predict more stable visual features by employing FV targets based on DINOv3 or V-JEPA 2, as these emphasize absolute position and task identity, which helps in maintaining performance consistency across different manipulation scenarios.

Bold header: Reshaped Attention for Task Relevance

The policy's attention maps will be steered toward relevant visual information, as evidenced by Figure 8 showing that both LA and FV supervision can reshape the policy’s attention toward task-relevant objects and interaction regions, supplying more task-relevant visual information for action prediction.