YOCO: You Only Calibrate Once! Fast Mocap Calibration for Dexterous Teleoperation

arXiv:2610.11657 · cs.RO · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa, and with me are Dev and Taro, guest researcher.

Dev: Today's paper: "YOCO: You Only Calibrate Once! Fast Mocap Calibration for Dexterous Teleoperation".

Rosa: The gist: YOCO presents a fast few-shot, fine-tuning-free calibration framework that corrects biased hand-pose streams from a small set of paired raw and target poses to improve downstream dexterous teleoperation.

Dev: First, who's behind it and why it matters.

Paper summary: Rosa: So we're looking at YOCO: You Only Calibrate Once! Fast Mocap Calibration for Dexterous Teleoperation, and the main thing here is tackling those biases in low-cost hand trackers that change from user to user and even from one recording session to another.

Dev: Yeah, it claims they have a fast few-shot calibration framework that corrects these biased hand-pose streams using just a small set of paired raw and target poses to improve dexterity in teleoperation.

Taro: So, the core idea seems to be moving away from traditional methods that learn glovesignal-to-pose mappings or correct IMU alignment, and instead focusing on calibrating the derived three dee hand-keypoint interface after a tracker has already produced its pose estimate <ref:2610.11657#pg1>.

Rosa: Right, so they formulate this few-shot calibration by conditioning a HyperNet on those paired examples to predict LoRA updates for a frozen MANO handestimation module. It's designed to be session-specific without needing per-session gradient optimization.

Dev: That sounds efficient, but I want to know how it actually runs in terms of the loop rate and latency since we're talking about real-time control systems here.

Taro: The formulation uses a Target network that maps the twenty-one three dee hand keypoints to MANO pose parameters, and its parameters are trained once on clean data and then kept fixed <ref:2610.11657#pg3>. Then the Hypernetwork takes the calibration set as input and outputs a set of low-rank LoRA matrices that get injected into that frozen target network.

Rosa: So, the learning objective is to minimize the expected joint-position error of this resulting calibrated estimator, encouraging it to infer mocap bias from those paired examples and produce updates that correct unseen noisy poses drawn from the same distribution.

Dev: They use synthetic biases in the MANO pose space—things like per-finger zero offsets, perjoint residual offsets, and global or finger gains—to train the hypernetwork to compensate for realistic mocap bias. That sounds like a good way to test robustness outside of just clean data.

Taro: The evaluation shows that YOCO substantially reduces hand-pose error on biased mocap streams relative to uncalibrated input, reducing the PA-MPJPE from sixteen point five five mm down to ten point four five mm on the augmented InterHand2 point 6M evaluation set, with the largest improvement seen on fingertip-sensitive metrics like Tip-PA dropping from thirty-seven point one five mm to twenty-three point two nine mm.

Rosa: And it's not just about accuracy; they claim YOCO outperforms optimization- and fine-tuning-based baselines, and that this advantage stays even when they limit the LoRA capacity across matched setups.

Dev: I see, so the speed aspect is important because deployment latency matters a lot in teleoperation. The paper mentions that on CPU, YOCO requires thirteen point one six seconds for calibration before real deployment, compared to maybe thirty-four to over two hundred seconds for gradient fine-tuning baselines.

Paper summary: Taro: That difference in time is significant because it means the calibration itself isn't taking a huge chunk of time on the machine that's running the robot control loop. Also, they found that YOCO achieves an overall success rate of ninety percent in teleoperation tasks, compared to sixty-five percent for LoRA-Finetune and thirty-five percent for the uncalibrated baseline.

Rosa: So, what does this mean for someone who just listens to the show? It suggests that even though we use low-cost trackers that are inherently biased, we can get a much more reliable hand state estimation by doing this one-time calibration using just a few paired poses.

Dev: It means we don't have to constantly re-optimize the entire model for every single user or every session; instead, this HyperNet generates updates on the fly based on that small calibration set. That cuts down on computation significantly.

Taro: The authors also pointed out a limitation: they focus heavily on pose calibration and leave the MANO shape estimation to a separate stage, and while the assumption is that bias is stationary across sessions, high-frequency drift on less stable hardware might need re-calibration or a temporal extension of the HyperNet.

Rosa: Exactly, so it's a strong tool for correcting session-stationary bias with minimal upfront work, but we still have to consider those cases where the tracking gets unstable over time.

Dev: We also see that the gains are consistent across different users in their dataset rather than being driven by just one specific operator. That suggests this method offers a more general improvement in reliability for the whole class of low-cost glove data they tested.

Taro: The results on contact-sensitive tasks were particularly telling, like Earbud Pinch Grasping rising from a one/ten success rate with LoRA-Finetune to an eight/ten success rate with YOCO, and Tissue Box Opening going from eight/ten to a full ten/ten.

Rosa: So the paper, YOCO: You Only Calibrate Once! Fast Mocap Calibration for Dexterous Teleoperation, presents a method where you use a small set of paired poses to condition a HyperNet that predicts LoRA updates for a frozen MANO estimator to correct hand pose biases quickly and robustly.

Dev: It's about coupling that sensor-independent pose interface with amortized, session-conditioned weight generation without needing per-session gradient optimization, which should be very appealing for deployment on real hardware.

Taro: The implication is that even with imperfect low-cost tracking, you can achieve much better dexterity in teleoperation tasks when you apply this calibration to correct the errors in the hand state estimation.

Rosa: We'll keep an eye on how it holds up outside of that single tensile glove evaluation, but for now, YOCO looks like a solid way to get those measurable gains on downstream manipulation quality and success rates.

Conclusion: Rosa: So we've looked at how YOCO uses just a few paired poses to correct those hand pose biases from low-cost trackers to make teleoperation better for dexterous tasks.

Dev: Yeah, it's about this hypernetwork generating updates for a frozen model instead of retraining the whole thing every time you get new data.

Rosa: The title itself, "You Only Calibrate Once!", really captures that idea of doing the heavy lifting upfront and then using it efficiently for everything else in a session.

Dev: And the authors are focused on making this calibration method fast enough to actually be useful in a real-time control loop, which is where my brain immediately goes.

Rosa: They're showing how this approach beats optimization and fine-tuning methods when it comes to accuracy on these specific contact tasks, like pinching things or opening boxes.

Dev: The numbers they give are pretty compelling, especially that jump in success rates from about sixty-five percent for other methods to ninety percent with YOCO.

Rosa: It means if you're building a system using cheaper tracking hardware, this calibration step is something you can do once and then rely on for much better performance down the line.

Dev: But we have to keep an eye on those caveats they mentioned regarding high-frequency drift, because that could be a problem if the hardware isn't super stable.

Rosa: Exactly, it's a solid tool for getting immediate gains in manipulation quality with less setup time, but we still have to figure out how long it holds up in the real world without constant re-calibration.

Dev: So what this means for you listening right now is that if you're working on a robotic system where hand tracking is noisy, there might be a way to get better results without spending all your time optimizing and retuning models for every single new recording session.

The University of Hong Kong

cs.RO

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: CoRL 2026

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: The gist: YOCO presents a fast few-shot, fine-tuning-free calibration framework that corrects biased hand-pose streams from a small set of paired raw and target poses to improve downstream dexterous

Key concepts

Few-shot Calibration
Instead of training a separate model for every user or session, YOCO uses a small set of paired raw and target poses to quickly learn how to correct biases. It treats the calibration process as generating 'low-rank' updates that specialize a main estimation module, allowing it to adapt rapidly without extensive retraining.
Hypernetwork
A Hypernetwork is a network whose weights are generated by another network. In YOCO, the calibration HyperNetwork takes the paired poses as input and outputs these low-rank LoRA matrices. This allows the system to dynamically generate session-specific correction parameters based on the observed calibration examples.
LoRA Updates
LoRA (Low-Rank Adaptation) updates are small, efficient adjustments made to a frozen neural network. YOCO predicts these updates from the calibration set, injecting them into the main hand estimation module. This allows for session-specific tuning of the model's behavior without needing to fine-tune or optimize the entire large model.
MANO Hand Estimation Module
This is a frozen component of YOCO that maps 3D hand keypoints to MANO pose parameters. The calibration process modifies this module using LoRA updates, rather than retraining it from scratch. This separation allows the system to focus solely on learning the bias correction while keeping the core shape estimation stable.

Terminology

Summary

The gist: YOCO presents a fast few-shot, fine-tuning-free calibration framework that corrects biased hand-pose streams from a small set of paired raw and target poses to improve downstream dexterous teleoperation.

Problem

Low-cost hand trackers often introduce biases that vary across users, devices, and recording sessions, leading to unreliable teleoperation <ref:2610.11657#pg3>. These biases are detrimental in dexterous manipulation where even small fingertip or joint offsets can alter contact patterns, pinch geometry, and retargeted robot motion <ref:2610.11657#pg10>. Prior hand-mocap calibration typically operates within a particular sensing pipeline, learning glovesignal-to-pose mappings, correcting IMU alignment and drift, or personalizing sensor-to-pose models <ref:2610.11657#pg3>. YOCO instead formulates few-shot calibration at the derived 3D hand-keypoint interface after a tracker has produced its pose estimate <ref:2610.11657#pg3>.

YOCO Framework

YOCO conditions a calibration HyperNet on paired examples and predicts LoRA-style updates for a frozen MANO handestimation module <ref:2610.11657#pg4>. Instead of optimizing a separate model for every operator or session, YOCO conditions a calibration HyperNet on the paired examples and predicts LoRA-style updates for a frozen MANO handestimation module <ref:2610.11657#pg3>. The formulation requires no access to device-specific raw signals; our real-device evaluation remains limited to one tensile glove <ref:2610.11657#pg3>.

Methodology

The calibration is formulated as the composition of two networks: a Target network and a Hypernetwork <ref:2610.11657#pg4>. The Target network maps 21 3D hand keypoints to MANO pose parameters, and its parameters ϕ are trained once on clean data and then held fixed <ref:2610.11657#pg5>. The Hypernetwork takes the calibration set as input and outputs a set of low-rank LoRA matrices ∆W, which are then injected into the frozen target network to form a session-specific calibrated estimator gϕ+∆W <ref:2610.11657#pg4>.

Training and Objective

The learning objective is to minimize the expected joint-position error of the resulting calibrated estimator, encouraging hψ to infer the underlying mocap bias from the paired calibration set and produce LoRA updates that correct unseen noisy poses drawn from the same distribution <ref:2610.11657#pg4>. The training proceeds in two stages: first, the target network gϕ is trained on clean data, then it is frozen and the hypernetwork hψ is trained to generate LoRA updates that compensate for realistic mocap bias <ref:2610.11657#pg5>. The augmentation strategy involves generating synthetic biases in the MANO pose space by combining several physically plausible error modes like per-finger zero offsets, perjoint residual offsets, and global/finger/joint gain errors <ref:2610.11657#pg14>.

Evaluation

Experiments evaluate YOCO across five research questions covering calibration accuracy, source of advantage, sample efficiency in K, transfer to real low-cost glove data, and downstream impact on dexterous teleoperation <ref:2610.11657#pg6>. Key results show that YOCO substantially reduces hand-pose error on biased mocap streams relative to uncalibrated input <ref:2610.11657#pg9>. Furthermore, YOCO outperforms optimization- and fine-tuning-based baselines, and the advantage persists at matched LoRA capacity <ref:2610.11657#pg9>. Downstream results indicate that YOCO achieves an overall success rate of 90% in teleoperation tasks <ref:2610.11657#pg10>.

Conclusion

YOCO reframes mocap calibration as feed-forward weight generation: a hypernetwork hψ reads a handful of paired poses and emits LoRA updates ∆W that specialize a frozen MANO estimator gϕ, avoiding per-session optimization of the estimator <ref:2610.11657#pg10>. The downstream results show promising teleoperation gains for the evaluated calibration-and-retargeting pipelines <ref:2610.11657#pg10>. The limitation noted is that YOCO focuses on pose calibration and leaves MANO shape estimation to a separate stage <ref:2610.11657#pg10>. The session-stationary bias assumption is consistent with sensor manufacturing tolerances of most commercial gloves, but highfrequency drift on less stable hardware would require re-calibration or a temporal extension of the HyperNet <ref:2610.11657#pg10>. The augmentation pipeline lets us scale training across diverse drift patterns at low cost, though covering high-frequency drift remains an open direction tied to sensor characterization <ref:2610.11657#pg10>.

How it works

The YOCO model architecture combines a hand-keypoint encoder and a MANO parameter decoder in the target network <ref:2610.11657#pg4>. The calibration HyperNet consists of three submodules: (1) hand-keypoint encoder, (2) calibration mapping, and (3) LoRA parameter decoder <ref:2610.11657#pg5>. The calibration mapping is a transformer block with RoPE positional embedding that models the mapping relationship across the paired examples <ref:2610.11657#pg5>.

Key Findings

**- YOCO reduces PA-MPJPE from 16.55 mm (no calibration) to 10.45 mm on the augmented InterHand2.6M evaluation set <ref:2610.11657#pg9>. This improvement is largest on fingertip-sensitive metrics (Tip-PA drops from 37.15 to 23.29 mm) <ref:2610.11657#pg9>. The results confirm that the HyperNet successfully infers session-level drift from only K = 4 paired examples and that the predicted LoRA updates ∆W correct unseen queries from the same session <ref:2610.11657#pg9>. YOCO achieves the best score on every precision metric while being substantially faster than every gradient-based baseline <ref:2610.11657#pg9>. The deploymentlatency gap is even larger on CPU, where YOCO requires 13.16 s versus 34.47–223.25 s for gradient fine-tuning baselines <ref:2610.11657#pg9>. The per-user standard deviations in Appendix C.5 indicate that this advantage is consistent across the four users in the dataset rather than driven by any single operator <ref:2610.11657#pg10>. The largest per-task gains appear on contact-sensitive tasks: Earbud Pinch Grasping rises from 1/10 (LoRA-Finetune) to 8/10 (YOCO), and Tissue Box Opening from 8/10 to 10/10 <ref:2610.11657#pg10>. The earbud task uses a middle-finger pinch, whereas the calibration poses include an index-finger pinch but no middle-finger pinch, suggesting transfer to an unobserved fingertip configuration <ref:2610.11657#pg10>. The paper demonstrates that the bias structure of a real glove is non-linear enough that linear correction in MANO θ space is actively harmful <ref:2610.11657#pg9>. The method performs well on real-world data, attaining the best score on every primary metric while requiring 6.73 s of conditioning versus 14.57 s for LoRA-Finetune and 13.04 s for Full-Finetune <ref:2610.11657#pg9>. The overall success rate achieved by YOCO is 90%, compared to 65% for LoRAFinetune and 35% for the uncalibrated baseline <ref:2610.11657#pg10>. This demonstrates that improved hand-state calibration translates into measurable gains on downstream dexterous teleoperation tasks <ref:2610.11657#pg10>. The longest trial row reports the longest observed trial time for each task <ref:2610.11657#pg10>. The average CPU calibration time required before real deployment is 13.16 s for YOCO <ref:2610.11657#pg9>.

Improvements for AI systems

  1. The YOCO framework enables fast few-shot, fine-tuning-free calibration by conditioning a HyperNet on a small set of paired raw and target poses to predict LoRA-style updates for a frozen MANO handestimation module. This allows per-user calibration to be an amortized, session-conditioned weight generation step while preserving the geometric prior of MANO, significantly improving downstream performance compared to uncalibrated input.

  2. The system can achieve superior accuracy by minimizing the expected joint-position error using the objective: min ψ ES, (xnq,xcq) L fψ(xnq; S), xcq, (6), which encourages the HyperNet to infer the underlying mocap bias from the paired calibration set and produce LoRA updates that correct unseen noisy poses drawn from the same distribution.

  3. The improved system provides rapid adaptation by generating session-specific corrections in a single feed-forward pass, as described by Figure 2, which results in a Calibrated Pose (Output) mapping that corrects the input Raw Mocap (Noisy) to achieve Better Estimation, leading to improvements like reducing PA-MPJPE from 16.55 mm to 10.45 mm on augmented data.

  4. The framework facilitates deployment on resource-constrained hardware by requiring only one forward pass of the HyperNet for calibration; this is demonstrated by YOCO achieving a CPU adaptation time of 13.16 s versus over 223.25 s for full gradient fine-tuning baselines, making it suitable for in-the-field teleoperation rigs without a dedicated GPU.

  5. The system enhances dexterity in teleoperation by improving task success rates; for instance, it increased the Earbud Pinch Grasping success rate from 1/10 (LoRA-Finetune) to 8/10 (YOCO), demonstrating that improved hand-state calibration translates into measurable gains on downstream dexterous teleoperation tasks.

Abstract

Dexterous teleoperation requires reliable human-hand state estimations. However, common low-cost motion-capture gloves and markerless trackers often exhibit biases that vary across users, glove fit, and recording sessions, degrading retargeting and demonstration quality. We present YOCO, a fast few-shot, fine-tuning-free calibration framework that corrects biased hand-pose streams from a small set of paired raw and target poses. Instead of optimizing a separate model for every operator or session, YOCO conditions a calibration HyperNet on the paired examples and predicts LoRA-style updates for a frozen MANO hand-estimation module, turning per-user calibration into a lightweight feed-forward adaptation step while preserving the geometric prior of MANO and the efficiency of a compact estimator. We train YOCO with synthetic drift augmentations on InterHand2.6M and evaluate on augmented InterHand sequences, offline real glove data, and dexterous teleoperation tasks. Across these settings, YOCO improves calibration efficiency, hand-state estimation quality and teleoperation performance compared with uncalibrated input and standard calibration baselines.

Sources

Related papers