GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation
cs.RO, cs.CV
Submitted: 2026-09-04
Updated: 2026-10-09
Comments: Technical report by the AgiBot Research Team. Project page: https://ge-act-v2.github.io/
Code: https://github.com/NVIDIA/Isaac-GR00T
Project page: https://ge-act-v2.github.io/Abstract
License: http://creativecommons.org/licenses/by/4.0/
The gist: World-action models (WAM) predict future states to guide robot actions, enabling learning from both action-free video and action-labeled interaction.
Abstract
World-action models (WAM) predict future states to guide robot actions, enabling learning from both action-free video and action-labeled interaction. Most inherit pretrained video generators, leaving WAM pretraining and scaling underexplored. We introduce Genie Envisioner Act 2.0 (GE-Act 2.0), a world-action model whose trainable generative and action components are all initialized from scratch on manipulation data. It combines a control-oriented autoencoder (CoAE), a single-step visual planner (SVP), and an inverse dynamics model (IDM). CoAE retains action- and instruction-relevant information under aggressive compression, while SVP produces a complete future state in one differentiable pass, so visual planning and inverse dynamics can be pretrained separately on complementary data. The components are then jointly trained with knowledge-aligned selective optimization (KASO), which reduces mismatched supervision by selecting only predicted futures judged behaviorally compatible with the recorded action. We evaluate pretrained checkpoints directly, without per-task fine-tuning, on 100 tasks across 20 manipulation skill groups with held-out scenes, backgrounds, lighting, and object instances. Scaling co-training data from 300 to 30,000 hours raises success from 17.1% to 44.1% on G1-OP and from 13.4% to 31.1% on G2-90D; despite comprising less than 2% of the co-training data, G2-90D improves by 17.7 points, suggesting cross-embodiment transfer. Gains span 19/20 and 18/20 skill groups, and skill-specific coverage strongly correlates with zero-shot out-of-distribution (OOD) success (Pearson r=0.80; Spearman rho=0.85). Under the same protocol, the model grounds object, color, shape, and position references in at least 90% of trials and follows explicit instructions even when they conflict with an already-committed behavior or a conventional scene association.
Sources
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- RT-1: Robotics Transformer for Real-World Control at Scale
- Genie: Generative Interactive Environments
- GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation
- Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models
- ABot-M0.5: Unified Mobility-and-Manipulation World Action Model
- RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation
- IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AI
- Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videos
- Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better
- LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models
- One Step Diffusion via Shortcut Models
- Mean Flows for One-step Generative Modeling
- Improved Mean Flows: On the Challenges of Fastforward Generative Models
- Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation
- Ctrl-World: A Controllable Generative World Model for Robot Manipulation
- LTX-Video: Realtime Video Latent Diffusion
- Learning Latent Dynamics for Planning from Pixels
- Dream to Control: Learning Behaviors by Latent Imagination
- Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies