AFFORD2ACT: How Far Can Sparse Keypoints Go for Robot Manipulation?

The gist: AFFORD2ACT introduces a language-conditioned affordance-guided keypoint extraction pipeline that selects sparse 2D keypoints from interaction-relevant regions without manual annotation, enabling efficient policy learning from a single external RGB view.

How it works

AFFORD2ACT is designed to overcome the challenge of extracting compact and expressive manipulation features by using language-conditioned affordance priors to localize where interaction should occur and extract a small set of semantic 2D keypoints that capture functionally relevant object parts The approach first filters candidate points to instruction-relevant regions, then preserves functional keypoint identity across object instances through correspondence, and finally adapts keypoint importance during control with gating that prioritizes keypoints for control

The pipeline consists of four main stages as outlined in the overview

1. Affordance Localization: This stage uses a pretrained One-Shot Open Affordance model to produce a heatmap indicating regions of the scene that correspond to the intended action, which is then converted into a binary mask

2. Reference Keypoint Extraction: Dense feature maps are computed using a DINO backbone on a reference frame, and attention is restricted to the Region of Interest (ROI) defined by the affordance mask to obtain reference anchor points

3. Cross-instance Correspondence: This step matches the reference anchor keypoints in a new frame by evaluating cosine similarity between the reference descriptor and per-pixel features, constrained to the affordance ROI, to ensure that the index refers to the same functional part across demonstrations

4. Temporal Keypoint Tracking: A transformer-based point tracker is used to track these keypoints over time, producing a temporally ordered set of keypoint locations that serves as input to the downstream policy

Policy Design

At each inference step, the policy transforms the unordered set of distilled keypoints into an action by first embedding each keypoint into a token using a learnable embedding function, forming a token set Next, contextualized token features are computed with a Transformer encoder block so that each keypoint is interpreted in the context of the others A gating network is introduced to assign a non-negative importance weight to each token, allowing the policy to down-weight task-irrelevant keypoints The resulting gated token features are aggregated into a compact scene embedding, which is then passed through a shared MLP action head that predicts the end-effector delta action and the gripper command

Evaluation and Contributions

The evaluation focuses on three claims: Keypoint Priors (E1), Keypoint Effectiveness (E2), and Generalization (E3)

• For Keypoint Priors, the method is compared against strategies like YOLO + DINO and SAM + DINO, finding that affordance-driven approaches do not depend on specific object labels and reach comparable performance to manual selection

• For Keypoint Effectiveness, the keypoint-based policy consistently outperforms RGB and RGB-D baselines across success rate, semantic trajectory metrics, and data scaling experiments

• For Generalization, AFFORD2ACT demonstrates strong transfer to novel object instances within seen categories and transfers to several new categories in test scenarios, particularly when objects share similar functional parts

Ablation Studies

Ablation studies highlight the necessity of key components in the pipeline

• Language Prompt Robustness showed that synonyms or related phrases produce similar affordance keypoints, indicating the model robustly captures intended action semantics

• The Gating Network was found to be critical; removing it significantly degraded performance from 86% to 52% in cluttered scenes

• The effective number of keypoints used by the policy, Keff, was typically much smaller than the total number of points K = 19 at any time step, implying that the gating network effectively attends to only a subset of the keypoints

Conclusion

AFFORD2ACT presents an imitation learning framework that uses language-conditioned affordance cues to distill a compact set of semantically grounded 2D keypoints for manipulation from a single external RGB view Across six real-robot tasks, AFFORD2ACT demonstrates data-efficient learning and robust performance under real-world variation, including unseen instances, novel categories with shared functional parts, background changes, and distractors This indicates that affordance-grounded keypoints can serve as an effective lightweight state representation for manipulation.

--- Page 1 ---

AFFORD2ACT: Affordance-Guided Automatic Keypoint Selection for Generalizable and Lightweight Robotic Manipulation Anukriti Singh1, Kasra Torshizi1, Khuzema Habib1, Kelin Yu1, Ruohan Gao1, Pratap Tokekar1 Abstract— Training vision-based manipulation policies that generalize beyond the training scenes remains challenging because dense visual observations entangle task-relevant cues with nuisance variation such as background, lighting, and instancespecific appearance. Keypoint representations offer a compact alternative, but automatically selecting task-relevant keypoints that transfer across objects and instructions is non-trivial. We present AFFORD2ACT (Affordance-Guided Keypoints to Act), which uses language-conditioned affordance priors to localize where interaction should occur and extract a small set of semantic 2D keypoints that capture functionally relevant object parts. Since keypoint relevance can change over the course of an interaction, we use a lightweight attention-and-gating policy to prioritize keypoints for control. Across six real-robot manipulation tasks, AFFORD2ACT achieves 82% success on held-out objects and is robust to clutter, distractors, and openvocabulary prompt variations. I. INTRODUCTION Learning effective robot manipulation policies hinges on overcoming a fundamental representation challenge: how can we extract manipulation-related features that are both compact and expressive? Dense visual inputs such as images [1], point clouds [2], or semantic features [3], [4] provide rich detail, but may burden a policy with extraneous information [5]. In contrast, 2D keypoints [6], [7] offer a sparse input representation that highlights only the essential features of objects while still being representative enough to learn a good policy. This sparsity directly enables a lightweight policy with far fewer, but more important, features than dense representations and supports effective real-time inference with lower computational demand. However, the real challenge is discovering these keypoints automatically and reliably: picking too few or irrelevant points can omit critical information, while relying on human insight for point selection does not scale. Automatic keypoint discovery for manipulation has recently gained momentum, with most methods falling into three paradigms. Motion-driven approaches identify keypoints as pixels (or point tracks) that move significantly when an object is manipulated, and then learn control from the resulting point tracks [8]–[10]. While effective when object motion is clean and informative, they can degrade in non-static scenes where background motion introduces spurious tracks, making it harder for the policy to remain reliable under dynamic distractors (e.g., a person operating in the same space as the manipulator). A second family of methods learns keypoints jointly with the policy, optimizing point selection end-to-end for task performance [11]. This can produce compact representations, but often couples the discovered points to the training data, making transfer to new objects or task intents less straightforward. Finally, a complementary direction seeks object-centric keypoints by extracting by grouping the semantic keypoints [12], [13]. These points can be helpful for consistent localization across frames and instances, but they are often agnostic to interaction: they describe where the object is and how it appears, rather than where the robot should act. Consequently, such keypoints may remain largely unchanged even when the intended use of the same object changes. In manipulation, however, relevance is inherently task-conditioned: the same tool supports multiple feasible interactions, and different instructions should highlight different regions (e.g.

1. Affordance-Guided Keypoint Extraction enables language-conditioned keypoint selection that selects sparse 2D keypoints from interaction-relevant regions without manual keypoint specification. This allows the AI system to automatically reduce a 224x224 image to a compact set of 19 keypoints, focusing only on functionally relevant object parts.

2. Correspondence-and-gating policy formulation preserves keypoint consistency across object instances while adapting keypoint relevance over the course of manipulation. This means the policy can dynamically re-weight keypoints over time, focusing on different interaction points at different stages of execution.

3. The system can achieve robust generalization by using affordance cues to guide distillation, ensuring the resulting representation is semantically grounded and directly useful for policy learning, which allows it to be robust to novel objects, clutter, and open-vocabulary prompts.

Vision-based manipulation policies are commonly learned from rich visual observations, typically using dense visual features extracted from full resolution images. It remains unclear how much of this visual information is really needed for effective policy learning. In this paper, we investigate the opposite end of the spectrum: how effective can manipulation policies be if they use as input only a compact visual representation. Sparse 2D keypoints provide a deliberately compact representation, giving the policy spatial information while removing direct access to appearance, texture, and background. We design AFFORD2ACT (Affordance-Guided Keypoints to Act) to study how much manipulation capability can be retained under this representation constraint. An off-the-shelf language-conditioned affordance model identifies the task-relevant region, from which up to 15 object keypoints are selected, together with 4 end-effector keypoints. The policy input is therefore reduced to only 19 2D keypoints (38 coordinates). A lightweight gating further decides which keypoints to pay attention to during execution. Across six real-robot manipulation tasks, this compact state outperforms the evaluated RGB, RGB-D, and dense-affordance behavior cloning baselines, degrades less as the number of demonstrations is reduced, and achieves 82% average success in held-out settings. Replacing learned gating with uniform weights reduces success from 92% to 52%. At the same time, the representation depends on reliable affordance localization and keypoint tracking, and remains vulnerable to incorrect masks, occlusion, tracking drift, and distractors.