Bridging Handheld and Teleoperated Supervision for Contact-Rich Manipulation via State-Gated Experts
cs.RO
Submitted: 2026-06-25
Updated: 2026-09-21
Comments: Project Page: https://nperi-rai.github.io/bridge-project/
License: http://creativecommons.org/licenses/by/4.0/
The gist: Handheld data collection systems, such as the Universal Manipulation Interface (UMI), enable scalable data collection across diverse environments but only capture observed actions rather than the
Terminology
Abstract
Handheld data collection systems, such as the Universal Manipulation Interface (UMI), enable scalable data collection across diverse environments but only capture observed actions rather than the desired actions executed by a robot controller. In contrast, teleoperation captures desired actions directly, but requires a robot in-the-loop. We revisit this trade-off through the lens of action validity across task phases. We observe that handheld trajectories provide valid supervision in tolerant, free-space phases, but can be poor supervision targets for contact-sensitive phases, requiring high stiffness to minimize tracking errors, resulting in large contact forces. We study the interaction between these two supervision types for contact-rich manipulation and find that training policies that combine handheld data with a small number of targeted teleoperated demonstrations provide an effective hybrid strategy. Specifically, rather than teleoperating the entire task, we only collect partial teleoperated demonstrations for task segments where base handheld policies fail. However, naively mixing handheld and teleoperated phase-specific data yields worse performance than training on handheld data alone. To address this mismatch between observed and desired supervision, we propose Bi-modal Routing for Imitation Data via Gated Experts (BRIDGE), a mixture of diffusion policy experts that routes between specialist task phase heads conditioned on the current robot state. Notably, our approach enables task-phase specific use of desired actions during contact sensitive segments and improves success rates over handheld-only baselines by up to 36.7% across three contact-rich manipulation tasks.
Sources
- Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots
- GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators
- Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
- Open X-Embodiment: Robotic Learning Datasets and RT-X Models
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
- Octo: An Open-Source Generalist Robot Policy
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- Human-in-the-Loop Imitation Learning using Remote Teleoperation
- Residual Policy Learning
- Compliant Residual DAgger: Improving Real-World Contact-Rich Manipulation with Human Corrections
- Force-Aware Residual DAgger via Trajectory Editing for Precision Insertion with Impedance Control
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
- Abstracting Robot Manipulation Skills via Mixture-of-Experts Diffusion Policies
- MoE-ACT: Scaling Multi-Task Bimanual Manipulation with Sparse Task-Conditioned Mixture-of-Experts Transformers
- DINOv2: Learning Robust Visual Features without Supervision
- Perceiver IO: A General Architecture for Structured Inputs & Outputs
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies