Human Demonstrations as Post-Training Data for Any Robot Body: Retargeting the Object, Not the Hand
cs.RO, cs.AI
Submitted: 2026-10-14
Updated: 2026-10-14
Comments: 16 pages, 17 figures, 7 tables, 2 algorithms; 56-page supplement as ancillary file. A subset of episodes with their per-robot outcomes: https://playground.dxtr.co
License: http://creativecommons.org/licenses/by/4.0/
The gist: Human demonstrations are today used largely for pretraining vision-language-action models, with little ability to use the same data for target-embodiment or task-oriented training, which scalable
Abstract
Human demonstrations are today used largely for pretraining vision-language-action models, with little ability to use the same data for target-embodiment or task-oriented training, which scalable deployment of robots in industrial and general environments depends on. A human demonstration recorded on a simple and scalable device such as a mobile phone can serve as post-training data for a robot whose body differs in structure and capability from the human who recorded it. This is made possible not by retargeting the hand's motion but the object's. A capture yields a body-free record: the object's pose over time, its contact intervals, and a reconstructed 3D object with provenance. Any robot, given as a URDF and a short descriptor, is solved against that record and scored in simulation against the demonstrated object motion as ground truth: the human-to-simulation leg of human-to-simulation-to-robot transfer, measured per robot. Each (robot, episode) pair gets one outcome: pass, failure with a named mode, or refusal, with its reason, before any motion. The method runs as a working system over a growing collection; the set analysed here is a sample of it: 42 captures solved on 8 robots, 157 pairs, of which 31.2% pass, 52.2% fail, 14.0% are refused and 2.5% are unauditable (kinematic feasibility, +/-50 mm / +/-15 deg, in simulation). Refusals held up: every pair the checks refuse on stored evidence yet ran anyway failed (13 of 13; 5 on a body whose public model welds its hand). A description file could not be shown to screen a robot: its outcomes were barely more predictable than guessing. No policy has yet been trained on the record. The record decides what a robot may post-train on; whether the policy then works on the real robot is the simulation-to-robot question, answered downstream.
Sources
- Dynamic Grasping with Reachability and Motion Awareness
- Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation
- EC-Flow: Enabling Versatile Robotic Manipulation from Action-Unlabeled Videos via Embodiment-Centric Flow
- X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real
- RoboPocket: Improve Robot Policies Instantly with Your Phone
- Video2Sim2Real: Full-Stack Autonomous Dexterous Skill Acquisition from a Single Human Video
- DITTO: Demonstration Imitation by Trajectory Transformation
- SPOT: SE(3) Pose Trajectory Diffusion for Object-Centric Manipulation
- PhysV2A: Reachability-Gated and Semantic-Mask-Constrained Feasibility Completion for Video-to-Robot Manipulation
- DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning
- EgoMimic: Scaling Imitation Learning via Egocentric Video
- Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
- OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitation
- Execution-Aware Pre-Execution Ranking for Grasp-Conditioned Robotic Placement
- MediaPipe: A Framework for Building Perception Pipelines
- Crossing the Human-Robot Embodiment Gap with Sim-to-Real RL using One Human Demonstration
- Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalization
- MobileEgo Anywhere: Open Infrastructure for long horizon egocentric data on commodity hardware
- What Enables In-Context Behavior Prompting for Manipulation?
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies