Same Goal, Different Words: Spatial Instruction Sensitivity in World-Action Models
cs.RO
Submitted: 2026-10-12
Updated: 2026-10-15
Comments: 18 pages, including appendices. Restored the four-page main-text layout and corrected the General Motors affiliation. Institutional disclaimer retained; scientific content unchanged
License: http://creativecommons.org/licenses/by/4.0/
The gist: World-action models should preserve the intended manipulation across equivalent descriptions of a spatial goal.
Abstract
World-action models should preserve the intended manipulation across equivalent descriptions of a spatial goal. We evaluate this consistency in 864 rollouts from three models across four RoboLab scenes, holding initial states and task goals fixed. Describing relations through the reference object rather than the object-to-move lowers the rate of reaching a stable placement by 12.2 percentage points on average. The effect of instruction rephrasing varies across spatial relations, so aggregate scores can hide sensitivity to wording. Standalone tests of the policies' language understanding identify relation reversals and show that adding images can disrupt otherwise correct text-only interpretations. Rollout analysis also distinguishes spatial placement from instruction following: moving the wrong object can satisfy the requested relation, and rates of stable placement fall from 43.2% at any time to 21.8% at episode end. These findings motivate a more thorough evaluation of world-action models' steerability, language understanding, and grounding. Evaluations should test consistency across equivalent instructions, verify that the intended object moved, and check whether the requested arrangement persists to the end of the rollout.
Sources
- A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation
- When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs
- LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models
- CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Models
- SG-WAM: Text-Grounded and Spatial-aware Semantic Guidance for World-Action Models
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies