Learning Transferable World-Action Models from Task-Paired Human Videos
cs.CV, cs.RO
Submitted: 2026-10-09
Updated: 2026-10-09
Comments: Project Page: https://watch-wam.github.io/
Project page: https://watch-wam.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: World-action models inherit broad visual and physical priors from video pre-training.
Abstract
World-action models inherit broad visual and physical priors from video pre-training. However, standard within-video next-frame prediction allows them to rely on appearance shared between the context and future, rather than learning interactions that transfer across scenes and embodiments. We introduce WATCH, a cross-video pre-training that learns from task-paired human demonstrations collected across diverse environments. Conditioned on the video and actions of one demonstration, the model jointly predicts another demonstration's actions and future frames from its current observation. Predicting one demonstration from another encourages the model to learn task-relevant interactions that transfer across scenes and demonstrators. Our pre-training shows that these capabilities can be learned from readily scalable human-human pairs alone, without requiring matched human-robot demonstrations. Analyses before robot post-training show improved motion alignment across embodiments and better action and video prediction on unseen robots, with representations closer to those learned from robot pairs. After robot post-training, WATCH improves MimicDroid in-context success from 40.9% to 68.3% with Cosmos Policy and from 59.8% to 77.7% with Cosmos3, with the largest gains on the unseen levels. Gains persist under context-free post-training across robot embodiments. WATCH outperforms within-video pre-training and co-training baselines, supporting cross-video prediction as a scalable approach to learning transferable world-action representations.
Sources
- Cosmos World Foundation Model Platform for Physical AI
- Wan: Open and Advanced Large-Scale Video Generative Models
- Fast-WAM: Do World Action Models Need Test-time Future Imagination?
- World Action Models are Zero-shot Policies
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data
- MimicDroid: In-Context Learning for Humanoid Robot Manipulation from Human Play Videos
- Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Time
- EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videos
- EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World
- Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos
- Cosmos 3: Omnimodal World Models for Physical AI
- HAND Me the Data: Fast Robot Adaptation via Hand Path Retrieval
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulation
- Emergence of Human to Robot Transfer in Vision-Language-Action Models
- In-Context Imitation Learning via Next-Token Prediction
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models