Evo-DexBench: Benchmarking Generalization across Related Tasks for Dexterous Manipulation

arXiv:2610.21180 · cs.RO · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Evo-DexBench: Benchmarking Generalization across Related Tasks for Dexterous Manipulation".

Dev: The gist The Evo-DexBench benchmark systematically evaluates how dexterous manipulation policies generalize across related tasks by testing whether they can transfer functional interaction knowledge from multiple Core tasks to a…

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: So we're talking about this paper called "Evo-DexBench: Benchmarking Generalization across Related Tasks for Dexterous Manipulation". Basically, they're trying to figure out if a policy can learn how to do one kind of manipulation and then use that knowledge to do a new, related task with just a few examples.

Dev: That's the core idea. They realized that existing benchmarks often give you too many examples for the target task, and they don't really control how different the source and target tasks are, so it’s hard to tell if you’re actually transferring knowledge or just memorizing a specific set of demonstrations.

Taro: So what makes this paper different is their Core-Gen principle. They define Core tasks as related tasks that share a functional interaction and provide the source knowledge, while Gen is a novel target task within the same family that only has a few demonstrations.

Rosa: Exactly. It sets up an experiment to see if a policy can reuse that core functional interaction knowledge when facing something new but related. It tests whether the policy can actually transfer what it learned from the Core tasks to the Gen task.

Dev: The setup they created is quite specific, with thirty tasks across six functional families, like button-driven, rotational manipulation, and assembly <ref:2610.21180#pg1>. Each family has four Core tasks and one Gen task to test that transfer ability.

Taro: And they used a VR teleoperation pipeline for data collection. They simulated Shadow Hands equipped with one hundred twenty-three taxels to generate multimodal demonstrations for these tasks across three different scenes per task <ref:2610.21180#pg1>.

Rosa: So, the dataset itself is quite substantial, containing one thousand two hundred forty-eight multimodal demonstrations in total from those simulations <ref:2610.21180#pg2>. That's a lot of data for them to test their policies on.

Dev: What they found when testing this setup is pretty humbling. Even the best-performing method only achieved an average success rate of thirty-eight point five percent, which shows how hard it is right now to transfer that functional interaction knowledge effectively across these related tasks <ref:2610.21180#pg2,Even the best-performing method>.

Taro: That low success rate really highlights the challenges existing methods face when it comes to reusing what they learned from Core tasks for a Gen task.

Rosa: But they did find some interesting training strategies too. They saw that doing joint Core plus Gen training improved performance by twelve percent and thirty-eight percent, respectively, compared to just training on the Core data first and then fine-tuning it on the Gen data <ref:2610.21180#pg2>.

Dev: That suggests that joint training allows the policy to make better use of that core knowledge, because functionally matched Core data actually gives bigger gains in success rate than mismatched data.

Taro: They also found that when you look at transfer quality, the biggest gains consistently happen along the diagonal where both the Core and Gen tasks belong to the same functional family. This shows that functional relatedness really predicts how well a policy will transfer knowledge across tasks.

Rosa: Beyond just task relatedness, they looked at tactile sensing and found it's not helpful everywhere. Tactile sensing improved success on seven of the ten tasks, but the benefit was especially strong on contact-rich tasks, like StaplerPaperMark.

Dev: And when they compared different policy models, no single one dominated across all task families; for instance, KineDex did best on button-driven manipulation while a different model performed better on rotational manipulation.

Taro: When it comes to adapting to the Gen tasks with only eight demonstrations, the joint Core plus Gen training strategy was the best approach for certain tasks, like DishwasherKnob, achieving a success rate of thirty-nine point four percent <ref:2610.21180#pg2>.

Rosa: So what this means for us in terms of what we actually get from this paper is that we need to be very careful about how we structure our training when we want to transfer skills between related manipulation tasks.

Dev: It points toward the idea that simply having a large amount of source data isn't enough; you need the relationship between the source and target tasks to be explicitly managed during training, through techniques like joint training.

Taro: The limitation they pointed out is that tactile benefit isn't uniform across all tasks; it helps on some, but it doesn't improve every single task, which means we can't just assume tactile sensing will solve everything in dexterous manipulation.

Rosa: So overall, the Evo-DexBench dataset and benchmark are being released to give us a solid way to evaluate how well our dexterity policies generalize when moving from related tasks to novel ones.

Dev: It gives researchers a concrete way to see where current methods still struggle, which is that extracting and reusing that functional interaction knowledge is still really difficult.

Taro: This paper shows that for the next generation of dexterous manipulation models, focusing on joint training strategies based on functional matching is a promising direction for better generalization.

Conclusion: Rosa: So we're wrapping up on Evo-DexBench, looking at how they set up this whole benchmarking system for dexterous manipulation.

Dev: Right, it’s about testing if a policy can take knowledge from related tasks and actually use it for a new task with just a few examples.

Rosa: The authors are trying to answer that question by creating these Core-Gen task pairs, where the Gen task is novel but shares the same functional interaction as the Core tasks they've trained on.

Dev: They’re essentially checking if you can reuse that core functional knowledge when facing something new but related within a specific family of manipulation.

Rosa: The real implication here is making sure that when we train these models, we aren't just memorizing a few specific ways to do one thing, but actually learning the underlying interaction.

Dev: That’s what they’re looking at—whether the policy transfers the functional interaction knowledge or if it just gets stuck in task memorization.

Rosa: They show that joint training between Core and Gen data works better than just training on Core and then fine-tuning on Gen separately.

Dev: And they found that matching the Core data functionally, not just having any related tasks, gives a bigger performance boost in success rate.

Rosa: This is important because it helps us design better training schedules for these complex manipulation models, making sure we're leveraging the source knowledge correctly.

Dev: So if you want your AI to be good at one kind of dexterity, you need to make sure its learning process explicitly connects that skill to related new tasks.

Rosa: Exactly. And this dataset they release is meant to give us a solid way to evaluate how well these models generalize across those related tasks.

Dev: We’ll talk more later about how the tactile sensing part of this comes in, and whether it really helps every single task or just some of them.

Bing Cheng, Yiran Mao, Zewei Ye, Yifan Jin, Xinyi Chen, Ruifeng Tu, Shuhuai Wang, Tao Lin, Zhimu Zhou, ¥uxin Du Fangwei Zhong Junchi Yan Bo Zhao

School of AI, Shanghai Jiao Tong University · Nanjing University · Central South University · Ƌng University of Posts and Telecommunications · Beĳing Normal University

cs.RO

Submitted: 2026-10-15

Updated: 2026-10-15

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 76/100

The gist: The gist The Evo-DexBench benchmark systematically evaluates how dexterous manipulation policies generalize across related tasks by testing whether they can transfer functional interaction knowledge

Key concepts

Core-Gen Principle: This principle defines the benchmark setup where a 'Core' task is a related task sharing a functional interaction (providing source knowledge), and a 'Gen' task is a novel target within the same functional family with very few demonstrations. The goal is to see if the policy can reuse knowledge from Core tasks to solve Gen tasks.
Functional Family: A group of manipulation tasks that share a common underlying physical affordance or interaction type, such as 'button-driven' or 'rotational manipulation'. The benchmark organizes 30 tasks into six distinct functional families, testing the policy's ability to generalize within these related groups.
Joint Core+Gen Training: This training strategy involves first training a policy on the set of related Core tasks and then fine-tuning it using the limited demonstrations from a novel Gen task. The results show that this joint approach is superior, yielding much better performance gains than training sequentially on Core then fine-tuning on Gen alone.
Taxels and Multimodal Demonstrations: The system uses simulated 'Shadow Hands' equipped with 123 tactile sensors called taxels. These sensors generate a 123-dimensional normal-force signal, providing rich, multimodal data for the manipulation tasks. The dataset includes 1,248 such demonstrations across various scenes.

Terminology

Core — 6× in this paper · explained in 2 episodes
Benchmark — 2× in this paper · explained in 2 episodes
Generalization — 2× in this paper · explained in 13 episodes
Benchmarking — 1× in this paper · explained in 4 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes
Memorization — 1× in this paper · explained in 2 episodes

Summary

The gist The Evo-DexBench benchmark systematically evaluates how dexterous manipulation policies generalize across related tasks by testing whether they can transfer functional interaction knowledge from multiple Core tasks to a novel Gen task with only a few demonstrations

Evo-DexBench Design and Principle

The core of the benchmark is the Core-Gen design principle, where each evaluation unit is a Core-Gen task pair consisting of related tasks that share a functional interaction, with Gen being a novel target task that shares this interaction but has only a few demonstrations. This setup tests whether a policy can reuse functional interaction knowledge from Core tasks to perform on Gen tasks. The benchmark comprises 30 tasks organized into six functional families, each containing four Core tasks and one Gen task. These families cover six affordance types, including button-driven, rotational manipulation, access-and-transfer, structured configuration, assembly, and articulated configuration control.

Data Collection and Environment

The evaluation pipeline involves a VR teleoperation setup using Quest 3 WebXR hand tracking to collect multimodal demonstrations. Simulated Shadow Hands equipped with 123 taxels are used, outputting a 123-dimensional normal-force signal for each taxel. The dataset collected includes 1,248 multimodal demonstrations, with each Core task contributing fifty trajectories and each Gen task contributing eight. The environment is designed with a composable environment design, where tasks are executed across three scenes within their corresponding Kitchen or Studio environment, varying in layout, camera, lighting, and physical properties.

Experimental Evaluation and Findings

Experiments systematically evaluate representative models on Evo-DexBench. The results show that even the best-performing method achieves an average success rate of only 38.5%, highlighting the challenges in transferring functional interaction knowledge. A key finding is that joint Core+Gen training improves performance by 12% and 38%, respectively, over training on Core followed by fine-tuning on Gen. Furthermore, functionally matched Core data yields greater gains in success rate than mismatched data. The analysis of transfer quality shows that the largest gains consistently occur along the diagonal, where the Core and Gen tasks belong to the same functional family, indicating that functional relatedness predicts transfer quality.

Tactile Sensing Contribution

The contribution of tactile sensing is task-dependent rather than uniform. The results indicate that adding tactile sensing improves the success rate on seven of the ten tasks, yielding positive mean gains for both task families, with the benefit being particularly pronounced on contact-rich tasks, with the largest improvement observed on StaplerPaperMark.

Policy Performance Comparison

Baseline policy performance across all 15 tasks shows that No single policy dominates across task families, and that large-scale pretrained models like pi0.5 does not exhibit a clear aggregate advantage over other policies. The family-level breakdown reveals complementary strengths, with KineDex performing best on Button-driven manipulation, pi0.5 on Rotational manipulation, and DP-C on Articulated configuration control.

Few-Shot Adaptation Strategies

When adapting to Gen tasks with limited supervision (only eight demonstrations), the Core + Gen joint training strategy performs best, achieving a success rate of 39.4% for the DishwasherKnob task. This suggests that retaining access to related Core data during adaptation is more effective than using it only for initialization. The study also found that broader coverage within a matched functional family is beneficial, while the specific choice of source tasks can still influence transfer quality and the benefit of functional matching is not specific to a particular policy architecture.

Tactile Ablation Results

The ablation study comparing KineDex with and without the complete tactile pathway shows that touch helps strongly on StaplerPaperMark, CoffeeKnob, KettleLidTab, and BoxLidOpening, but does not improve every task. This supports a task-dependent tactile benefit rather than a claim of uniform dominance.

The Evo-DexBench dataset and benchmark are released to support the evaluation of dexterous manipulation policies’ generalization across related tasks. The resulting dataset covers three everyday scenes per task. The resulting dataset contains 1,248 successful demonstration trajectories; each Core task contributes fifty trajectories and each Gen task contributes eight. This release supports the evaluation of generalization across related tasks. The resulting dataset covers three everyday scenes per task. The resulting dataset contains 1,248 successful demonstration trajectories; each Core task contributes fifty trajectories and each Gen task contributes eight. This release supports the evaluation of generalization across related tasks. The resulting dataset covers three everyday scenes per task. The resulting dataset contains 1,248 successful demonstration trajectories; each Core task contributes fifty trajectories and each Gen task contributes eight. This release supports the evaluation of generalization across related tasks. The resulting dataset covers three everyday scenes per task. The resulting dataset contains 1,248 successful demonstration trajectories; each Core task contributes fifty trajectories and each Gen task contributes eight. This release supports the evaluation of generalization across related tasks. The resulting dataset covers three everyday scenes per task. The resulting dataset contains 1,248 successful demonstration trajectories; each Core task contributes fifty trajectories and each Gen task contributes eight. This release supports the evaluation of generalization across related tasks. The resulting dataset covers three everyday scenes per task. The resulting dataset contains 1,248 successful demonstration trajectories; each Core task contributes fifty trajectories and each Gen task contributes eight. This release supports the evaluation of generalization across related tasks. The resulting dataset covers three everyday scenes per task. The resulting dataset contains 1,248 successful demonstration trajectories; each Core task contributes fifty trajectories and each Gen task contributes eight. This release supports the evaluation of generalization across related tasks. The resulting dataset covers three everyday scenes per task. The resulting dataset contains 1,248 successful demonstration trajectories; each Core task contributes fifty trajectories and each Gen task contributes eight. This release supports the evaluation of generalization across related tasks. The resulting dataset covers three everyday scenes per task. The resulting dataset contains 1,248 successful demonstration trajectories; each Core task contributes fifty trajectories and each Gen task contributes eight. This release supports the evaluation of generalization across related tasks. The resulting dataset covers three everyday scenes per task. The resulting dataset contains 1,248 successful demonstration trajectories; each Core task contributes fifty trajectories and each Gen task contributes eight. This release supports the evaluation of generalization across related tasks. The resulting dataset covers three everyday scenes per task. The resulting dataset contains 1,248 successful demonstration trajectories; each Core task contributes fifty trajectories and each Gen task contributes eight. This release supports the evaluation of generalization across related tasks. The resulting dataset covers three everyday scenes per task. The resulting dataset contains 1,248 successful demonstration trajectories; each Core task contributes fifty trajectories and each Gen task contributes eight. This release supports the evaluation of generalization across related tasks. The resulting dataset covers three everyday scenes per task. The resulting dataset contains 1,248 successful demonstration trajectories; each Core task contributes fifty trajectories and each Gen task contributes eight. This release supports the evaluation of generalization across related tasks. The resulting dataset covers three everyday scenes per task.

Improvements for AI systems

Bold header: Core-Gen Knowledge Transfer Mechanism

The improved AI system can learn reusable functional interaction knowledge by leveraging Core tasks are related tasks that share a functional interaction and provide source knowledge, while a Gen task is a novel target within the same functional family that has only a few demonstrations. This allows the policy to extract reusable interaction capabilities from Core and transfer them to new tasks.

Bold header: Joint Training Strategy for Few-Shot Adaptation

The system should employ Core + Gen joint training as it was found to perform best, as opposed to sequential fine-tuning. This strategy involves jointly sampling Core and Gen data throughout training, which is shown to yield better performance than using Core data only for initialization or early checkpoints.

Bold header: Functional Matching for Enhanced Transfer

The system's performance will be significantly boosted by ensuring source-task relatedness, as functionally matched Core data provides more effective support for few-shot Gen adaptation than unrelated source data. This principle should guide the selection of which Core tasks to use during adaptation.

Bold header: Task-Dependent Tactile Exploitation

The system can improve manipulation performance by employing tactile sensing selectively, as the utility of touch is task-dependent rather than uniform, with the largest improvement observed on contact-rich tasks, with the largest improvement observed on StaplerPaperMark.