ZeroDex: Zero-Shot Long-Horizon Dexterous Manipulation via Multi-View 3D-Grounded VLM Reasoning

arXiv:2606.19340 · cs.RO · Submitted 2026-06-17 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Hi, I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "ZeroDex: Zero-Shot Long-Horizon Dexterous Manipulation via Multi-View 3D-Grounded VLM Reasoning".

Dev: Here's the gist of the paper. ZeroDex presents a zero-shot framework for long-horizon dexterous manipulation that grounds language instructions into executable 3D task plans from calibrated multi-view RGB images.

Rosa: The gist is that ZeroDex uses a vision-language model (VLM) to produce reference-frame task grounding and primitive-level 2D keypoints, then lifts them into 3D via multiview fusion to support both pick-and-place and tool-use tasks through modular reasoning and reusable action primitives.

Dev: How it works The framework is built on a modular design that decouples semantic reasoning from physical execution, leveraging the competence of modern VLMs to answer most manipulation sub-questions zero-shot The process begins with reference-frame semantic grounding where a VLM selects a reference view and infers the manipulation mode, producing a grounding tuple gz=(r, z, gz) Following grounding is the multi-view fusion-based 3D lifting stage where view-dependent 2D predictions are lifted into consistent 3D quantities How it works (Continued) The lifted keypoints are then used for object-centric atomic action alignment for tool-use tasks For dexterous execution, a Bag of Atomic Actions is used for tool-use behaviors Grasp candidates are then generated using a cylindrical-template sampler for handle-like affordances or an optimization-based generator Physical feasibility is ensured by applying collision-aware position refinement to grounding points Finally, an off-the-shelf arm-hand motion generator tracks the desired trajectory Tobj to resolve kinematic and collision constraints How it works (Continued) Experimental results show improved 3D grounding accuracy and execution reliability over single-view RGB-D grounding and fine-tuned VLA baselines How it works (Continued) The framework's limitations include performance being bounded by the reasoning reliability of the underlying 2D VLM The system's success in long-horizon tasks is enabled by its ability to recover from intermediate failures through closed-loop retry mechanisms The overall system successfully couples VLM-based task planning and multi-view 3D grounding with reusable atomic action primitives and closed-loop verification to enable zero-shot execution in unseen scenes The system's capabilities include target grounding amidst distractors, spatial-relation reasoning, affordance-aware tool use, and long-horizon sequencing<ref:2606.

Title and authors: Rosa: Moving on to the authors and the title of ZeroDex: Zero-Shot Long-Horizon Dexterous Manipulation via Multi-View three dee-Grounded VLM Reasoning <ref:2606.19340#pg1,ZeroDex: Zero-Shot Long-Horizon Dexterous Manipulation via Multi-View 3D-Grounded>. This paper brings together several researchers from Seoul National University, including Jisoo Kim, Sangwon Baik, Taeksoo Kim, Sungjoo Kim, Junyoung Lee, Mingi Choi and Hanbyul Joo.

Dev: The title really nails the core concept: zero-shot manipulation using a multi-view approach grounded in three dee reasoning driven by a vision-language model <ref:2606.19340#pg1>. It’s about making complex physical tasks doable without needing massive amounts of task-specific training data.

Taro: Zero-shot is key here because it means the system doesn't need to be trained from scratch for every new object or tool; it just needs the instruction and the images.

Rosa: It’s about grounding those instructions into executable three dee task plans from calibrated multi-view RGB images, moving beyond just recognizing objects in a single frame <ref:2606.19340#pg1,instructions into executable 3D task plans from calibrated multi-view RGB images>.

Dev: The authors argue that a modular design is more efficient because instead of learning one end-to-end policy, you separate the semantic reasoning, handled by the VLM, from the physical execution.

Taro: That separation is what allows them to leverage the fact that modern VLMs already answer many of those manipulation sub-questions zero-shot without needing a custom policy.

Rosa: They use a vision-language model to produce reference-frame task grounding and primitive-level 2D keypoints, which they then lift into three dee via multiview fusion <ref:2606.19340#pg1,to produce reference-frame task grounding and primitive-level 2D keypoints>.

Dev: The core innovation there is how they combine triangulation of view-wise VLM groundings with reference-view ray voting to search along a semantic camera ray for geometrically consistent candidates across neighboring views.

Taro: That voting process is what allows them to produce three dee keypoints that are geometrically consistent, which the paper says supports both pick-and-place and tool-use tasks <ref:2606.19340#pg1,both pick-and-place and tool-use>.

Rosa: And for tool use, it means they can retrieve an object-centric atomic action corresponding to the inferred skill category and align its stored 6D tool trajectory to the scene <ref:2606.19340#pg1,retrieve an object-centric atomic action corresponding to the inferred skill category>.

Dev: It’s a modular approach where the VLM does the high-level planning and decomposition into simple atomic tasks like pick and move, which are then executed by reliable motion primitives.

The paper's summary: Rosa: To summarize what ZeroDex does, it takes a language instruction and calibrated multi-view observations to produce an executable three dee task plan <ref:2606.19340#pg1>. It starts with a vision-language model producing reference-frame task grounding and primitive-level 2D keypoints <ref:2606.19340#pg1,reference-frame task grounding and primitive-level 2D keypoints>.

Dev: Then, they take those 2D predictions and lift them into consistent three dee quantities using multiview fusion, which includes that triangulation and ray voting we talked about earlier <ref:2606.19340#pg1>.

Taro: That lifting process is what creates the reliable three dee keypoints necessary to support pick-and-place tasks as well as more complex tool-use maneuvers <ref:2606.19340#pg1>.

Rosa: For tool use, they use those three dee keypoints to generate affordance-aware dexterous grasps and then execute plans through reusable action primitives that are driven by the VLM reasoning <ref:2606.19340#pg1>.

Dev: Essentially, the paper is arguing that modern VLMs are competent enough to answer most of the complex sub-questions manipulation requires zero-shot, so we don't need to train a new policy from scratch.

Taro: The system decomposes a complex task into a sequence of simple atomic tasks such as pick and move, and it executes each one with a reliable primitive controller.

Rosa: This modular design means that because most manipulation tasks can be expressed as compositions of these atomic tasks, this small library of primitives covers a broad range of tasks.

Dev: The main implication is that we don't need to spend huge amounts of data training a new policy when we see something new in the world or need to use a different tool.

The paper's improvements: Rosa: The authors highlight several specific improvements they made over previous work, starting with grounding accuracy improvement through multi-view fusion. They combine "triangulation of view-wise VLM groundings with reference-view ray voting" to produce reliable three dee grounding under occlusion and partial visibility <ref:2606.19340#pg1,triangulation of view-wise VLM groundings with reference-view ray voting>.

Dev: That combination of triangulation and ray voting is what they use to improve three dee grounding by searching along a semantic camera ray for geometrically consistent candidates across neighboring views <ref:2606.19340#pg1,along a semantic camera ray for geometrically consistent candidates across neighboring views>.

Taro: That search finds those candidates that are robust even when the object is partially hidden in one view, which addresses a big problem with single-view RGB-D grounding.

Rosa: Then there’s robust task planning via closed-loop verification, which enables long-horizon manipulation through closed-loop status verification and replan, allowing recovery from intermediate failures or object displacement within the retry budget.

Dev: That closed loop mechanism is what allows the system to recover from things that go wrong mid-task, like an object suddenly moving or a scene changing unexpectedly.

Taro: That capability means the system can handle novel objects and tool use tasks in novel scenes by allowing it to recover from those intermediate failures.

Rosa: They also enhanced dexterous grasp generation by estimating a task-conditioned grasp affordance region, which they then filter for collision feasibility using trajectory-level inverse-kinematics and collision feasibility over the full tool-use trajectory.

Dev: That filtering step is crucial because it ensures that the generated grasps are not just plausible but also physically feasible when considering the entire tool-use motion.

Conclusion: Rosa: So to wrap up, ZeroDex shows how to couple VLM-based task planning and multi-view three dee grounding with reusable atomic action primitives and closed-loop verification to enable zero-shot execution in unseen scenes <ref:2606.19340#pg1>.

Dev: The overall implication is that this system successfully couples VLM-based task planning and multi-view three dee grounding with those atomic action primitives, providing a way to achieve zero-shot execution by handling the complexity through modular reasoning <ref:2606.19340#pg1>.

Taro: For me, the success in long-horizon tasks is really about its ability to recover from intermediate failures through closed-loop retry mechanisms, which shows it can handle the messy reality of interaction.

Rosa: And remember, a key limitation they state is that performance remains bounded by the reasoning reliability of the underlying 2D VLM, meaning we still need better vision models to make this work perfectly <ref:2606.19340#pg1>.

Dev: So ZeroDex demonstrates a powerful framework for long-horizon dexterity, but it shows that even with limitations in the vision model, we can achieve zero-shot execution in unseen scenarios by relying on those closed-loop verification and atomic action primitives.

Taro: It opens up possibilities for autonomous systems to handle complex physical tasks where they don't have perfect pre-training data for every possible scenario.

Rosa: That’s the story of ZeroDex, using multi-view three dee grounding and VLM reasoning to build a system that plans and executes in the real world with less upfront training <ref:2606.19340#pg1>.

Jisoo Kim, Sangwon Baik, Taeksoo Kim, Sungjoo Kim, Junyoung Lee, Mingi Choi, Hanbyul Joo

Seoul National University

cs.RO

Submitted: 2026-06-17

Updated: 2026-10-20

Project page: https://jlogkim.github.io/zerodex

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: ZeroDex presents a zero-shot framework for long-horizon dexterous manipulation that grounds language instructions into executable 3D task plans from calibrated multi-view RGB images.

Key concepts

Reference-Frame Semantic Grounding
A vision-language model selects a specific view from the input images to establish a reference frame. It then infers the overall manipulation mode (e.g., pick-and-place or tool use) based on this view, creating a tuple that defines where the system should focus its attention for task planning.
Multi-View Fusion for 3D Lifting
The system takes 2D predictions from different camera views and combines them into a single, consistent 3D representation. This is achieved by lifting view-dependent keypoints into a shared spatial coordinate system, allowing the model to understand objects and actions in three dimensions rather than just flat images.
Closed-Loop Status Verification
To handle long tasks reliably, the framework includes a mechanism to check the status of the manipulation after each step. If an intermediate failure occurs (like object displacement), it triggers a replan within a set budget, allowing the system to recover and complete complex, multi-step instructions zero-shot.

Terminology

Summary

ZeroDex presents a zero-shot framework for long-horizon dexterous manipulation that grounds language instructions into executable 3D task plans from calibrated multi-view RGB images. The gist is that ZeroDex uses a vision-language model (VLM) to produce reference-frame task grounding and primitive-level 2D keypoints, then lifts them into 3D via multiview fusion to support both pick-and-place and tool-use tasks through modular reasoning and reusable action primitives.

How it works

The framework is built on a modular design that decouples semantic reasoning from physical execution, leveraging the competence of modern VLMs to answer most manipulation sub-questions zero-shot

The process begins with reference-frame semantic grounding where a VLM selects a reference view and infers the manipulation mode, producing a grounding tuple gz=(r, z, gz)

Following grounding is the multi-view fusion-based 3D lifting stage where view-dependent 2D predictions are lifted into consistent 3D quantities

How it works (Continued)

The lifted keypoints are then used for object-centric atomic action alignment for tool-use tasks

For dexterous execution, a Bag of Atomic Actions is used for tool-use behaviors Grasp candidates are then generated using a cylindrical-template sampler for handle-like affordances or an optimization-based generator Physical feasibility is ensured by applying collision-aware position refinement to grounding points Finally, an off-the-shelf arm-hand motion generator tracks the desired trajectory Tobj to resolve kinematic and collision constraints

How it works (Continued)

Experimental results show improved 3D grounding accuracy and execution reliability over single-view RGB-D grounding and fine-tuned VLA baselines

How it works (Continued)

The framework's limitations include performance being bounded by the reasoning reliability of the underlying 2D VLM

The system's success in long-horizon tasks is enabled by its ability to recover from intermediate failures through closed-loop retry mechanisms The overall system successfully couples VLM-based task planning and multi-view 3D grounding with reusable atomic action primitives and closed-loop verification to enable zero-shot execution in unseen scenes The system's capabilities include target grounding amidst distractors, spatial-relation reasoning, affordance-aware tool use, and long-horizon sequencing<ref:2606.

Improvements for AI systems

  1. Grounding accuracy improvement through multi-view fusion: The system improves 3D grounding by combining triangulation of view-wise VLM groundings with reference-view ray voting, which produces reliable 3D grounding under occlusion and partial visibility.

  2. Robust task planning via closed-loop verification: The framework enables long-horizon manipulation through closed-loop status verification and replan, enabling zero-shot execution on unseen objects and tool-use tasks in novel scenes by allowing the system to recover from intermediate failures, object displacement, or newly observed scene changes within the retry budget.

  3. Enhanced dexterous grasp generation: The system generates feasible grasps by estimating a task-conditioned grasp affordance region, which is filtered for collision feasibility using trajectory-level inverse-kinematics and collision feasibility over the full tool-use trajectory.

  4. Improved tool use execution: Tool use is supported by retrieving an object-centric atomic action corresponding to the inferred skill category and aligning its stored 6D tool trajectory to the scene, ensuring that the motion is anchored correctly based on inferred skills.

Abstract

We present ZeroDex, a zero-shot framework for long-horizon dexterous manipulation that grounds language instructions into executable 3D task plans from calibrated multi-view RGB images. Our framework builds on the grounding ability that vision-language models (VLMs) already exhibit in 2D and lifts their predictions into 3D through multi-view fusion, which extends VLM grounding beyond image-space keypoints to the metric 3D quantities that manipulation actually requires: contact points, placement targets, and tool-motion anchors. Because execution is driven by these 3D groundings rather than a learned policy, the system preserves the VLM's open-world generalization, without collecting demonstrations or post-training for a specific task or embodiment. The VLM decomposes a complex instruction into a sequence of atomic tasks, each executed by a primitive from a small reusable library that covers manipulation tasks including pick-and-place and tool-use: for tool-use, we retrieve an object-centric atomic action corresponding to the inferred skill category and align its stored 6D tool trajectory to the grounded keypoints; for dexterous execution, we expand the lifted grasp keypoint into a task-conditioned grasp affordance region and generate feasible grasp-motion pairs with an arm-hand motion generator. Real-world experiments show improved 3D grounding accuracy and higher task success than zero-shot visual-prompting and constraint-based baselines, both of which ground from a single RGB-D view, and our demonstration-free system outperforms VLA policies fine-tuned on 100 task demonstrations. We further demonstrate long-horizon manipulation, where execution verification and replanning recover from intermediate failures, and show that the system generalizes to unseen objects and tool-use tasks in novel scenes.

Sources

Related papers