Pixel2Catch: Multi-Agent Sim-to-Real Transfer for Agile Manipulation with a Single RGB Camera

The gist: Pixel2Catch proposes an RGB-only robotic catching system that infers object motion from pixel-level features rather than explicit 3D position estimation, enabling successful sim-to-real transfer.

How it works

1. The system is inspired by human visual perception, where object motion is inferred from pixel-level features in image space rather than metric 3D coordinates <ref:2602.22733#pg2>.

2. To extract these cues, the work employs the Segment Anything Model 2 (SAM 2) [10], which provides robust object segmentation and enables the extraction of pixel-level information capturing relative changes in image space position and scale <ref:2602.22733#pg2>.

3. The proposed pixel-based feature vector is defined as zpixel ∈ R 6 = cx, cy, ∆cx, ∆cy, ∆w, ∆h (1) <ref:2602.22733#pg5>. This vector includes the object’s image space center coordinates, width and height <ref:2602.22733#pg5>.

4. Temporal differences are incorporated into zpixel to capture motion dynamics, as the paper states that we incorporate temporal differences between consecutive timesteps into zpixel <ref:2602.22733#pg5>.

5. A heterogeneous Multi-Agent Reinforcement Learning (MARL) framework is designed to control the high-DoF system by decomposing it into independent agents: a robot arm and a multi-fingered hand <ref:2602.22733#pg2>.

6. The arm agent focuses on approaching the object with the end-effector, while the hand agent is trained to stably grasp the object without dropping it <ref:2602.22733#pg6>.

7. The policies are trained using MultiAgent Proximal Policy Optimization (MAPPO) [11] within a Centralized Training with Decentralized Execution (CTDE) paradigm <ref:2602.22733#pg5>.

System Components and Setup

The physical setup utilizes a 6-DoF robot arm, specifically the Universal Robots UR5e, mounted at a height of 0.81 m <ref:2602.22733#pg6>. The Allegro Hand is attached to the arm, with three joints fixed to simplify control and maintain Z (m) <ref:2602.22733#pg6>. A single RealSense D435 camera provides a global view, mimicking human visual perception by using only RGB images without depth information <ref:2602.22733#pg6>.

The simulation environment is developed using the NVIDIA Isaac Lab framework [36], employing a USD model identical to the real hardware and modeling the camera as a pinhole camera with the same field of view <ref:2602.22733#pg6>. To improve robustness, we randomize the object’s initial state at the beginning of each episode, including its position, orientation, linear velocity, and mass <ref:2602.22733#pg6>.

Learning Framework and Reward Design

The problem is formulated as a Multi-Agent Markov Decision Process (MAMDP), where each agent maximizes its expected discounted return E[PT k=0 γ k r i k] <ref:2602.22733#pg5>. Separate reward functions are designed for the arm and hand policies to encourage role-specific behavior <ref:2602.22733#pg6>.

The reward for the arm policy is formulated as Rarm t = rtime + rpalm dist + λsucc1succ + λapp1app − λfail(1drop + 1coll) − λactaarm t2 <ref:2602.22733#pg6>. The reward for the hand policy is Rhand t = 1/5(rpalm dist + X i∈F r i dist) + λsucc1succ − λfail(1drop + 1coll) − λactahand t2 <ref:2602.22733#pg6>. The term rdist is defined as the temporal difference in Euclidean distance between the robot link and the object <ref:2602.22733#pg6>.

Results and Sim-to-Real Transfer

The proposed method consistently outperforms baselines across both simulation and real-world environments, achieving a tracking rate of approximately 70% and a catching success rate of around 50% in real-world experiments <ref:2602.22733#pg8>. The performance gain over Single-Agent Reinforcement Learning (S-A RL) stems from the task decomposition into specialized roles, allowing for role-specific reward functions and reducing unnecessary coupling between reaching and grasping objectives <ref:2602.22733#pg7>.

Crucially, the system demonstrates strong sim-to-real transfer capability because all policies are transferred directly from simulation to the real robot without any additional real-world fine-tuning <ref:2602.22733#pg8>. The paper concludes that this approach enables agile and stable catching of human-thrown objects using only RGB visual input <ref:2602.22733#pg2>.

Future Work

The current study is limited to a single-arm robotic setup, and future work plans include extending the proposed framework to bimanual robotic platforms to enable more stable and robust catching of objects with diverse sizes and geometries. The paper also notes that performance degradation in real-world settings for baseline methods is attributed to differences in object trajectory distributions between simulation and the real world <ref:2602.22733#pg8>.

--- Page 1 ---

Pixel2Catch: Multi-Agent Sim-to-Real Transfer for Agile Manipulation with a Single RGB Camera

Seongyong Kim, Junhyeon Cho, Kang-Won Lee, and Soo-Chul Lim

Dongguk University

Deploy in Real-world Train in Simulation Time Monocular RGB Camera Thrown Object by Human Multi-joint Robot Without 3D Position (0.1, 0.2, 1.5) RGB Image Object Perception using Pixel-wise Features Cx, Cy (W, H) C W H Learning Catching Policy Fig. 1: We propose Pixel2Catch, an RGB-only robotic catching system without explicit 3D position estimation. The system consists of a robot arm equipped with a multi-fingered hand and a single RGB camera. Inspired by human visual perception, object motion is inferred from pixel-level features in image space rather than metric 3D coordinates. Policies trained in simulation are transferred directly to the real robot without fine-tuning. Abstract— To catch a thrown object, a robot must be able to perceive the object’s motion and generate control actions in a timely manner. Rather than explicitly estimating the object’s 3D position, this work focuses on a novel approach that recognizes object motion using pixel-level visual information extracted from a single RGB image. Such visual cues capture changes in the object’s position and scale, allowing the policy to reason about the object’s motion. Furthermore, to achieve stable learning in a high-DoF system composed of a robot arm equipped with a multi-fingered hand, we design a heterogeneous multiagent reinforcement learning framework that defines the arm and hand as independent agents with distinct roles. Each agent is trained cooperatively using role-specific observations and rewards, and the learned policies are successfully transferred from simulation to the real world. I. INTRODUCTION When humans catch a thrown object, they do not rely on explicit 3D position values or precise metric representations of the object <ref:2602.22733#pg2>. Instead, they perceive the object’s motion by observing relative changes in its apparent position and size over time, and generate catching movements based on these visual cues. This highlights that dynamic catching can be achieved through the interpretation of visual motion patterns, rather than explicit estimation of the object’s 3D state <ref:2602.22733#pg2>. However, most prior studies on dynamic object manipulation have relied on the motion capture systems [1]–[5] or depth sensors [6]–[9] to estimate the object’s 3D position <ref:2602.22733#pg2>. While such sensing modalities enable explicit geometric measurements, the performance of the resulting control models is strongly influenced by sensor quality <ref:2602.22733#pg2>. In particular, accurately estimating 3D object positions in real-world environments is often more challenging than in simulation, which leads to a significant sim-to-real gap when transferring learned policies <ref:2602.22733#pg2>.

1. Improved object motion perception by using pixel-level features extracted from a single RGB image instead of explicit 3D position estimation, which makes performance less sensitive to sensor accuracy. This allows the system to recognize object motion using pixel-level visual information extracted from a single RGB image.

2. Implementation of a heterogeneous multiagent reinforcement learning framework that decomposes the robotic system into independent agents for the arm and hand, allowing them to be trained cooperatively with role-specific observations and rewards. This enables specialized control, where the arm focuses on positioning and the hand focuses on grasping.

3. Development of a state space that concatenates sequential observations over two consecutive timesteps for each agent, providing temporal context necessary for inferring motion dynamics directly from relative visual changes. This helps policies infer motion dynamics directly from relative visual changes.

4. Design of role-specific reward functions, such as those defined in equation (2) and (3), to train the arm policy to focus on approaching the thrown object with the end-effector and the hand policy to focus on stably grasp[ing] the object without dropping it. This clear signal helps stabilize learning compared to a unified reward.

5. Integration of system identification prior to training, where joint-level dynamic parameters in simulation are optimized based on real robot data, which minimizes the sim-to-real gap. This ensures that policies learned in simulation are directly deployed on the physical robot without additional fine-tuning.

6. Introduction of domain randomization for both observations and actions during training, including random noise perturbations, to enhance robustness against real-world variations. This augmentation accounts for segmentation uncertainty caused by real-world visual variations and helps the proposed method achieve robust performance under real-world conditions.

To catch a thrown object, a robot must be able to perceive the object's motion and generate control actions in a timely manner. Rather than explicitly estimating the object's 3D position, this work focuses on a novel approach that recognizes object motion using pixel-level visual information extracted from a single RGB image. Such visual cues capture changes in the object's position and scale, allowing the policy to reason about the object's motion. Furthermore, to achieve stable learning in a high-DoF system composed of a robot arm equipped with a multi-fingered hand, we design a heterogeneous multi-agent reinforcement learning framework that defines the arm and hand as independent agents with distinct roles. Each agent is trained cooperatively using role-specific observations and rewards, and the learned policies are successfully transferred from simulation to the real world.