ULTRA: Unified Multimodal Control for Autonomous Humanoid Whole-Body Loco-Manipulation

arXiv:2603.03279 · cs.RO, cs.CV · Submitted 2026-03-03 · Read on arXiv

cs.RO, cs.CV

Submitted: 2026-03-03

Updated: 2026-10-17

Comments: IROS 2026 Best Application (ICROS) and Mobile Manipulation (OMRON Sinic X) Paper Awards Finalist, Project Page: https://ultra-humanoid.github.io/

Project page: https://ultra-humanoid.github.io

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Distillation — 26× in this paper · explained in 2 episodes
• Robustness — 8× in this paper · explained in 5 episodes
• Transformer — 7× in this paper · explained in 2 episodes
• Latent Space — 5× in this paper · explained in 7 episodes
• Point Clouds — 5× in this paper · explained in 2 episodes
• Regularization — 5× in this paper · explained in 3 episodes
• Reinforcement Learning — 5× in this paper · explained in 4 episodes
• Alignment — 4× in this paper
• Fidelity — 4× in this paper · explained in 2 episodes
• ACM Transactions on Graphics — 3× in this paper
• Distribution Shift — 3× in this paper · explained in 10 episodes
• Embeddings — 3× in this paper · explained in 2 episodes
• Generalization — 3× in this paper · explained in 13 episodes
• Unified Framework — 3× in this paper · explained in 5 episodes
• Ablation — 2× in this paper · explained in 1 episode
• Actor — 2× in this paper · explained in 3 episodes
• Consistency — 2× in this paper · explained in 3 episodes
• Data Augmentation — 2× in this paper · explained in 8 episodes
• Fine-Tuning — 2× in this paper · explained in 5 episodes
• Foundation Model — 2× in this paper · explained in 3 episodes
• Action Matching Loss — 1× in this paper · explained in 1 episode
• Deep Reinforcement Learning — 1× in this paper · explained in 3 episodes
• Latent Representation — 1× in this paper · explained in 2 episodes
• Multimodal Learning — 1× in this paper · explained in 2 episodes
• Physical Constraints — 1× in this paper · explained in 7 episodes
• Regret — 1× in this paper · explained in 4 episodes
• Reinforcement Learning (RL) — 1× in this paper · explained in 25 episodes
• Sample Efficiency — 1× in this paper · explained in 7 episodes
• Sim-to-Real Transfer — 1× in this paper · explained in 2 episodes
• Tokenization — 1× in this paper · explained in 2 episodes

Sources

• Retargeting Matters: General Motion Retargeting for Humanoid Motion Tracking
• HOMIE: Humanoid Loco-Manipulation with Isomorphic Exoskeleton Cockpit
• GMT: General Motion Tracking for Humanoid Whole-Body Control
• DemoHLM: From One Demonstration to Generalizable Humanoid Loco-Manipulation
• ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills
• OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning
• ExBody2: Advanced Expressive Humanoid Whole-Body Control
• DreamControl: Human-Inspired Whole-Body Humanoid Control for Scene Interaction via Guided Diffusion
• BFM-Zero: A Promptable Behavioral Foundation Model for Humanoid Control Using Unsupervised Reinforcement Learning
• BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion
• SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control
• StyleLoco: Generative Adversarial Distillation for Natural Humanoid Robot Locomotion
• SPIDER: Scalable Physics-Informed Dexterous Retargeting
• DemoDiffusion: One-Shot Human Imitation using pre-trained Diffusion Policy
• Proximal Policy Optimization Algorithms
• Adversarial Locomotion and Motion Imitation for Humanoid Policy Learning
• ULC: A Unified and Fine-Grained Controller for Humanoid Loco-Manipulation
• PhysHSI: Towards a Real-World Generalizable and Natural Humanoid-Scene Interaction System
• HDMI: Learning Interactive Humanoid Whole-Body Control from Human Videos
• InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactions

Related papers

• FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
• MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
• RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
• ARC: A Reasoning Recipe for Robot Foundation Models
• HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
• APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies