GROW squared: Grounding Which and Where for Robot Tool Use

arXiv:2606.30632 · cs.RO, cs.AI, cs.CV · Submitted 2026-06-29 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "GROW squared: Grounding Which and Where for Robot Tool Use".

Dev: The gist: GROW2 introduces a novel approach for open-world affordance grounding in robot tool use by decomposing the problem into semantic selection and geometric grounding using object parts,

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So we're looking at a paper called "GROW squared: Grounding Which and Where for Robot Tool Use." It sounds like they're trying to figure out how robots can use tools they haven't explicitly been trained on, which is a big deal for open-world robotics.

Dev: Yeah, it tackles the challenge of open-world affordance grounding, which is essentially figuring out which object to use as a tool and exactly where on that object the robot should interact with it. It tries to do this without relying on huge datasets for every single possible tool and action.

Taro: I'm curious about what makes it different from existing methods because a lot of them assume you already know the exact tool beforehand, which is not realistic in a cluttered space.

Rosa: Exactly, that’s where they introduce this two-level decomposition using object parts as the main abstraction to split the grounding process. It separates picking the right tool and figuring out where to grasp it into two distinct steps.

Dev: So at the semantic level, they use vision-language models or VLMs to do the reasoning part, which means interpreting a natural language instruction like "cut the cake."

Taro: And then at the geometric level, they use vision foundation models to take those selected parts and ground them into precise three dimensional affordance regions from just one RGB-D image. That seems like a powerful way to handle ambiguity.

The paper's summary: Rosa: The core idea here is that by splitting the problem this way, they can leverage different foundation models for generalization instead of needing one massive model trained on every possible tool and action.

Dev: They use the VLM to select a suitable object as the tool and then identify task-relevant parts on both that tool and the target object. That handles the "Which" part of the problem, figuring out what to use and where to look.

Taro: And then they take those selected parts, say a screwdriver handle and a piece of wood, and use vision foundation models to ground them into three dee affordance regions from a single view <ref:2606.30632#pg1>. That addresses the "Where" part.

Rosa: They state that this decomposition eliminates the need for large-scale affordance annotations and lets the system generalize to open-category objects, which is what they call GROW2, or GROW squared.

Dev: They show that this approach allows them to produce complete and function-faithful three dee affordance regions from just a single view of an observation, which is something multi-view methods struggle with because they are inefficient for real deployment <ref:2606.30632#pg1>.

Taro: So the implication is they can potentially get a complete understanding of how to use any object as a tool based on its parts and the task instruction, without needing massive datasets for that specific interaction.

The paper's improvements: Rosa: One major improvement is that they move away from needing substantial three dee training data for traditional methods and instead rely on this part representation to decompose the grounding process <ref:2606.30632#pg1>.

Dev: They also address the fact that most prior work assumes you have a complete three dee observation, which isn't realistic, so GROW2 tackles that by using a single-view RGB-D image as input <ref:2606.30632#pg1>.

Taro: They handle occlusion challenges by combining 2D part grounding with three dee reconstruction <ref:2606.30632#pg1>. They reconstruct a high fidelity mesh, render it from multiple viewpoints, segment the parts in each view using SAM3, and then fuse those masks into complete three dee affordance regions <ref:2606.30632#pg1>.

Rosa: To make that geometric grounding more robust when the VLM might struggle with text queries, they use a vision-language model as a fallback to translate complex text queries into bounding boxes when SAM3 might not understand them.

Dev: And for the final step in finding the actual region, after getting those 2D segmentations, they back-project them onto the reconstructed mesh and use DBSCAN to cluster the points and keep only the largest cluster as their predicted affordance region <ref:2606.30632#pg1>.

Conclusion: Rosa: So, to wrap up on "GROW squared: Grounding Which and Where for Robot Tool Use," this paper presents a method that grounds affordances by decomposing the problem into semantic selection and geometric grounding using object parts.

Dev: It shows that by using VLMs for semantic selection and vision foundation models for geometric grounding, they can create complete three dee affordance regions from a single view observation <ref:2606.30632#pg1>.

Taro: The real world experiments on a Franka arm show that this method works reliably even with noisy real-world depth observations because it computes the affordance regions from the reconstructed mesh through multi-view rendering.

Rosa: The limitation they state is that failures can still happen across all fifty real-world trials, and tool and part selection accounts for the largest share of those failures. Plus, processing an entire scene takes about sixteen point six seconds, which points to a need to distill this into something much lighter.

Dev: That latency from querying VLMs and running inference on foundation models makes dynamic manipulation challenging right now. But overall, it's a zero-shot method for grounding affordances that can effectively support downstream robot tool use.

National University of Singapore · Massachusetts Institute of Technology

cs.RO, cs.AI, cs.CV

Submitted: 2026-06-29

Updated: 2026-10-19

Comments: CoRL 2026

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: The gist: GROW2 introduces a novel approach for open-world affordance grounding in robot tool use by decomposing the problem into semantic selection and geometric grounding using object parts,

Key concepts

Semantic Selection: This is the initial step where a vision-language model (VLM) uses commonsense knowledge to decide which tool and which specific parts of an object are relevant for the given task. It's about understanding what needs to be done and identifying potential tools and their necessary components based on language instructions.
Geometric Grounding: This second step takes the selected parts from the semantic level and maps them into precise 3D affordance regions in space. Vision foundation models are used for this, reconstructing a 3D shape from a single image and then using multiple views to create accurate masks for where the robot can actually grasp.
Part Representation: The core idea is to represent objects not just as whole items but as collections of distinct parts. By identifying these parts, the system can select tool components and target object features independently, making it easier to generalize its understanding to new objects.
Affordance Grounding: This refers to determining where a robot can physically interact with an object—where it can grasp or use something. GROW2 aims to predict these 3D regions of interaction from just one view, enabling the robot to plan actions like grasping based on predicted physical possibilities.

Terminology

Grounding — 4× in this paper · explained in 3 episodes
Benchmark — 2× in this paper · explained in 2 episodes
Abstraction — 1× in this paper · explained in 1 episode
Foundation Model — 1× in this paper · explained in 3 episodes
Foundation Models — 1× in this paper · explained in 7 episodes
Generalization — 1× in this paper · explained in 13 episodes
Vision-Language Models (VLMs) — 1× in this paper · explained in 6 episodes
Vision-language models — 1× in this paper · explained in 2 episodes
Zero-Shot Generalization — 1× in this paper · explained in 5 episodes

Summary

The gist: GROW2 introduces a novel approach for open-world affordance grounding in robot tool use by decomposing the problem into semantic selection and geometric grounding using object parts, enabling generalization across open-category objects.

How it works

GROW2 decomposes open-world affordance grounding into two levels: semantic selection and geometric grounding<ref:2606.30632#pg2> At the semantic level, GROW2 leverages the reasoning capabilities of vision-language models (VLMs) to select an appropriate tool and task-relevant parts<ref:2606.30632#pg2> At the geometric level, GROW2 grounds the selected parts into 3D affordance regions using vision foundation models<ref:2606.30632#pg2> This decomposition eliminates dependence on large-scale affordance annotations and lets GROW2 generalize to open-category objects<ref:2606.30632#pg2>

Tool and Part Selection

GROW2 starts with a cluttered scene of multiple objects and a task specified with a language instruction l<ref:2606.30632#pg5> Instead of finding an appropriate object as the tool solely through geometric features, GROW2 selects the tool in semantic space with the commonsense knowledge of VLMs<ref:2606.30632#pg6> Specifically, GROW2 first extracts the names of all objects and their parts as candidates<ref:2606.30632#pg5> After that, it performs tool and part selection conditioned on the task, the observation, and the extracted object-part candidates<ref:2606.30632#pg5> The process involves extracting object names using a VLM and then querying the VLM to generate a set of part candidates Pi for each object oi<ref:2606.30632#pg5> Task-relevant parts are categorized as (1) a part pG on the tool, indicating where the robot should grasp oA, and (2) a part pA on the tool oA and a part pB on the target object oB<ref:2606.30632#pg5> This selection is guided by in-context learning using examples to specify expected input-output format and provide chain-of-thought guidance<ref:2606.30632#pg5>

3D Affordance Grounding

The geometric level involves grounding the selected parts into 3D affordance regions using vision foundation models<ref:2606.30632#pg6> Given a single-view RGB-D observation, GROW2 reconstructs a high-fidelity object mesh and registers it to the observed scene<ref:2606.30632#pg6> To address occlusion challenges, GROW2 combines 2D part grounding with 3D reconstruction<ref:2606.30632#pg6> This involves rendering the registered mesh from multiple viewpoints, segmenting the selected parts in each view using SAM3, and fusing the resulting masks into complete 3D affordance regions<ref:2606.30632#pg6> For reliability, a VLM is used as a fallback to translate complex text queries into bounding boxes when SAM3 may fail to understand them<ref:2606.30632#pg6> Finally, the resulting 2D segmentations are back-projected onto the reconstructed mesh Mc, and DBSCAN is applied to cluster the points and retain only the largest cluster as the predicted affordance region<ref:2606.30632#pg6>

Experiments

GROW2 was evaluated on existing benchmarks like AGD20K and PIAD, where it achieved superior performance across most metrics In the simulation benchmark, GROW2 achieved the highest success rate across all five task types in SAPIEN 3 Real-world experiments on a Franka arm also showed that GROW2 can be deployed reliably in real-world settings, achieving performance comparable to simulation Specifically, in the real world, GROW2 is robust to noisy real-world depth observations because it computes affordance regions from a reconstructed mesh through multi-view rendering

Limitations

Failure cases across all 50 real-world trials of GROW2 arise across four stages, with tool and part selection accounting for the largest share of failures Efficiency is a concern because processing an entire scene takes approximately 16.6 seconds, highlighting the need to distill GROW2 into a lightweight affordance prediction model The high latency of querying VLMs and running inference with vision foundation models makes it challenging for the robot to perform dynamic manipulation

Conclusion

In this paper, GROW2 is introduced as a zero-shot method for grounding affordances in robot tool use By decomposing the problem with a part representation, GROW2 reformulates open-world affordance grounding as part grounding GROW2 can predict complete and functionally faithful 3D affordance regions from a single-view observation Experiments in both simulation and the real world show that the affordance regions predicted by GROW2 can effectively support downstream robot tool use

References

[1] T. Fitzgerald, A. Goel, and A. Thomaz Modeling and learning constraints for creative tool use in Frontiers in Robotics and AI, 8:674292, 2021

[40] J. J. Gibson The theory of affordances Hilldale, USA, 1(2):67–82, 1977

[45] G. Li V Jampani D Sun and L Sevilla-Lara Locate Localize and transfer object parts for weakly supervised affordance grounding in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023

[50] Y Yang W Zhai H Luo Y Cao J Luo and Z-J Zha Grounding 3d object affordance from 2d interactions in images in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023

[51] O Simeoni H V Vo M Seitzer F Baldassarre M Oquab C Jose V Khalidov M Szafraniec S Yi M Ramamonjisoa et al. Dinov3 arXiv preprint arXiv:2508.10104, 2025

[66] W Liu J Mao J Hsu T Hermans A Garg and J Wu Composable part-based manipulation in Conference on Robot Learning, 2023

[72] S M LaValle and J J Kuffner Jr. Randomized kinodynamic planning The International Journal of Robotics Research, 20(5):378–400, 2001

[55] F Xiang Y Qin K Mo Y Xia H Zhu F Liu M Liu H Jiang Y Yuan H Wang L Yi A X Chang L J Guibas and H Su SAPIEN A simulated part-based interactive environment in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020

[68] W Gao R Tedrake kpam-sc Generalizable manipulation planning using keypoint affordance and shape completion in IEEE International Conference on Robotics and Automation, 2021

[73] R B Rusu N Blodow and M Beetz Fast point feature histograms fpfh for 3d registration in IEEE International Conference on Robotics and Automation, 2009

[56] M Sundermeyer A Mousavian R Triebel and D Fox Contact-graspnet Efficient 6-dof grasp generation in cluttered scenes in IEEE International Conference on Robotics and Automation, 2021

[41] N Carion L Gustafson Y-T Hu S Debnath R Hu D Suris C Ryali K V Alwala H Khedr A Huang et al. Sam 3 Segment anything with concepts arXiv preprint arXiv:2511.16719, 2025

[42] X Chen F-J Chu P Gleize K J Liang A Sax H Tang W Wang M Guo T Hardin X Li et al. Sam 3d 3dfy anything in images arXiv preprint arXiv:2511.

Improvements for AI systems

A decomposed reasoning pipeline leveraging Vision-Language Models (VLMs) for semantic selection followed by vision foundation models for geometric grounding will enable zero-shot generalization over open-category objects by leveraging a series of foundation models and generalize to open-category objects.
The system can perform open-world affordance grounding by decomposing the process into two levels: semantically, using VLMs to select an appropriate tool and task-relevant parts, and geometrically, using vision foundation models to ground the selected parts into precise 3D affordance regions.
The robot can recover function-faithful 3D affordances from a single view by combining 2D part grounding with 3D reconstruction, which involves reconstructing a high-fidelity object mesh, rendering it from multiple viewpoints, segmenting parts in each view, and fusing the masks into complete 3D affordance regions.
Tool selection can be made controllable and interpretable by using in-context learning by including a small set of tool and part selection examples in the prompt, which allows for controllable and interpretable tool and part selection for tool use.
The system can support downstream manipulation through the generation of informative priors, specifically by deriving end-effector key poses based on the predicted affordance regions, which guides grasp pose sampling and object-level key poses using geometric constraints defined over these regions.