RoboAlign-R1: Distilled Multimodal Reward Alignment for Robot Video World Models
cs.RO, cs.AI
Submitted: 2026-05-05
Updated: 2026-09-27
Code: https://github.com/Alexander-wu/RoboAlign_R1
License: http://creativecommons.org/licenses/by/4.0/
The gist: Existing robot video world models are typically trained with low-level objectives such as reconstruction and perceptual similarity, which are poorly aligned with the capabilities that matter most for
Terminology
Abstract
Existing robot video world models are typically trained with low-level objectives such as reconstruction and perceptual similarity, which are poorly aligned with the capabilities that matter most for robot decision making, including instruction following, manipulation success, and physical plausibility. They also suffer from error accumulation in long-horizon autoregressive prediction. We present RoboAlign-R1, a framework that combines reward-aligned post-training with stabilized long-horizon inference for robot video world models. We construct RobotWorldBench, a benchmark of 10,000 annotated video-instruction pairs collected from four robot data sources, and train a multimodal teacher judge, RoboAlign-Judge, to provide fine-grained six-dimensional evaluation of generated videos. We then distill the teacher into a lightweight student reward model for efficient reinforcement-learning-based post-training. To reduce long-horizon rollout drift, we further introduce Sliding Window Re-encoding (SWR), a training-free inference strategy that periodically refreshes the generation context. Under our in-domain evaluation protocol, RoboAlign-R1 improves the aggregate six-dimension score by 10.1% over the strongest baseline, including gains of 7.5% on Manipulation Accuracy and 4.6% on Instruction Following; these ranking improvements are further supported by an external VLM-based cross-check and a blinded human study. Meanwhile, SWR improves long-horizon prediction quality with only about 1% additional latency, yielding a 2.8% gain in SSIM and a 9.8% reduction in LPIPS. Together, these results show that reward-aligned post-training and stabilized long-horizon decoding improve task consistency, physical realism, and long-horizon prediction quality in robot video world models.
Sources
- Qwen2.5-VL Technical Report
- Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imagination
- RT-1: Robotics Transformer for Real-World Control at Scale
- World Models
- TD-MPC2: Scalable, Robust World Models for Continuous Control
- Vid2World: Crafting Video Diffusion Models to Interactive World Models
- WorldModelBench: Judging Video Generation Models As World Models
- Evaluating Real-World Robot Manipulation Policies in Simulation
- Robot Learning from a Physical World Model
- Deep multi-scale video prediction beyond mean square error
- Video Diffusion Alignment via Reward Gradients
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- BridgeData V2: A Dataset for Robot Learning at Scale
- BeamVQ: Beam Search with Vector Quantization to Mitigate Data Scarcity in Physical Spatiotemporal Forecasting
- BeamVQ: Aligning Space-Time Forecasting Model via Self-training on Physics-aware Metrics
- RLVR-World: Training World Models with Reinforcement Learning
- Efficient Streaming Language Models with Attention Sinks
- PhyCritic: Multimodal Critic Models for Physical AI
- VideoGPT: Video Generation using VQ-VAE and Transformers
- Learning Interactive Real-World Simulators
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies