TrackDeform3D: Markerless and Autonomous 3D Keypoint Tracking and Dataset Collection for Deformable Objects

arXiv:2603.17068 · cs.CV, cs.RO · Submitted 2026-03-17 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "TrackDeform3D: Markerless and Autonomous 3D Keypoint Tracking and Dataset Collection for Deformable Objects".

Jane: Here's the gist of the paper. The gist: TrackDeform3D presents an affordable and autonomous framework for collecting 3D datasets of deformable objects using only RGB-D cameras by identifying 3D keypoints and robustly tracking their trajectories, incorporating motion consistency constraints to produce temporally smooth and geometrically coherent data Deformable objects such as wire harnesses or fabrics are ubiquitous in homes and factories, yet remain challenging for robots to manipulate.

Tom: Because these objects possess infinitely many degrees of freedom, directly modeling their full state is intractable.

Jane: Researchers therefore often approximate their state with structured 3D representations such as keypoints or meshes, which provide a compact yet informative description of their geometry and topology.

Tom: Large-scale, high-quality keypoint and mesh trajectory datasets of deformable objects could significantly advance robotic manipulation.

Title and authors: Tom: So we're looking at a paper called TrackDeformthree dee: Markerless and Autonomous three dee Keypoint Tracking and Dataset Collection for Deformable Objects <ref:2603.17068#pg1,TrackDeform3D: Markerless and Autonomous 3D Keypoint Tracking and Dataset Collection for Deformable>. I mean, "markerless" is the key here, because usually tracking these things requires you to stick physical markers on them, which is labor-intensive and messy.

Jane: Right, so if you don't need those physical markers to track a wire harness or a piece of fabric, that's a big deal because it opens up data collection ways that are way less painful for researchers and companies.

Lu: What makes this paper interesting is that they aren't just showing how to track one object; they are proposing an entire framework for autonomous collection using only standard RGB-D cameras.

Meng: Autonomous collection sounds good on paper, but what does "autonomous" actually mean in practice? Does it mean it works perfectly every time without human intervention during setup?

Tom: Well, the core idea is that the system can identify three dee keypoints and robustly track them using just RGB-D cameras, which solves a huge bottleneck we've seen <ref:2603.17068#pg1>.

Jane: And what they claim to solve is getting high-quality data for things like wire harnesses or fabrics where you can't just model the whole thing because it has infinite degrees of freedom.

Lu: They are proposing a system that identifies these keypoints and then enforces geometric consistency, which means if you collect data over different sessions, the keypoints stay in roughly the same place.

Meng: Consistency across sessions is hard to get without some kind of fixed reference point or something, right?

Tom: Exactly. They use anchor points for that purpose, which they say constrains the solution space and stabilizes where those keypoints are supposed to be >

The paper's summary: Jane: So, what is the actual process described in TrackDeformthree dee? It sounds like it’s a multi-step pipeline for taking video and turning it into these consistent three dee point clouds <ref:2603.17068#pg1>.

Tom: Basically, they start with an input of an RGB-D video sequence from time zero to T. The first big step is lifting that depth video to get a point cloud, and then they do some segmentation to figure out which parts are the deformable object itself >

Lu: They use point cloud differencing for this segmentation, where they filter out static background points before applying something called DBSCAN to cluster the remaining points into a segmented point cloud >

Jane: Then comes the initialization phase, where they classify the object type based on its local geometry and then determine anchor points and warm-start positions >

Tom: The real heavy lifting is in the problem formulation at each time step t, where they minimize an energy function to encourage uniform spacing among neighboring keypoints until it reaches an equilibrium >

Meng: Uniform spacing sounds like a lot of math to solve in real time. How are they managing that optimization loop without it taking forever?

Jane: They solve this constrained optimization problem by using an anchor set A to stabilize the solution space and ensure the keypoint configuration is stable across different collection sessions >

Tom: And for tracking, they re-detect those anchor points every frame, and then they solve that same optimization recursively, warm-started from the previous time step's solution >

Lu: Plus, they apply a temporal moving average filter to suppress high-frequency jitter in the resulting trajectories to make them smooth and temporally consistent >

The paper's improvements: Jane: So what are the actual improvements they point out over existing methods? It sounds like they’re focusing on initialization and tracking consistency.

Tom: They claim their two contributions are twofold, first an affordable tracking system that enforces geometric consistency for robust initialization and tracking across multiple collection sessions >

Lu: And the second is this large-scale dataset of 1D and 2D deformable objects, comprising six objects and totaling one hundred ten minutes of trajectory data for training and benchmarking deformable object models > <ref:2603.17068#pg2,objects and totaling 110 minutes of trajectory data for training and benchmarking>

Meng: A dataset of that size is huge. But what about the quality? Are these trajectories as clean as you'd expect from an autonomous system?

Jane: The paper shows they achieved lower Absolute edge length error, reported as RMSE in millimeters, and Chamfer distance compared to baselines like CDCPD2 and SpatialTrackerV2 >

Tom: And they also did ablation studies showing that both the anchor points and the edge length constraints are necessary during tracking; removing either one causes keypoints to drift or lose consistent spacing >

Lu: That’s important because it confirms that those specific components aren't just tacked on; they are essential for keeping the data geometrically accurate >

Conclusion: Jane: So to wrap up, what does TrackDeformthree dee actually mean for the field of modeling deformable objects? It seems like a lot of consistency achieved through careful optimization.

Tom: It means we can finally have a large-scale resource, which is six objects and one hundred ten minutes of data, annotated with keypoint trajectories and topology information > <ref:2603.17068#pg2>

Lu: This dataset addresses the scale and diversity gaps left by prior work, especially by including branched deformable objects which are crucial for things like wire harness modeling >

Meng: For deployment at scale, I’m concerned about the reliance on those anchor points being reliably detected every single frame. If those fail, the entire tracking system collapses >

Jane: That's a fair concern. But overall, TrackDeformthree dee successfully enforces anchor, edge length, and point cloud projection constraints to produce consistent and geometrically accurate keypoint trajectories across diverse object categories like DLOs, BDLOs, and cloth >

Tom: The work confirms the contribution of each optimization component through those ablation studies validating the necessity of anchor and edge length constraints for accurate and stable tracking >

Lu: It’s a solid pipeline that moves the goalpost on how we build these representations for complex geometries >

Meng: It's a big dataset, which is what matters most right now if we want to train models that actually work in the real world >

Yeheng Zong, Yizhou Chen, Alexander Bowler, Chia-Tung Yang, Ram Vasudevan

University of Michigan

cs.CV, cs.RO

Submitted: 2026-03-17

Updated: 2026-10-21

Comments: Accepted to the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2026

Project page: https://roahmlab.github.io/trackDeform3D-core-tracking

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 83/100

The gist: The gist: TrackDeform3D presents an affordable and autonomous framework for collecting 3D datasets of deformable objects using only RGB-D cameras by identifying 3D keypoints and robustly tracking

Key concepts

Anchor Points: These are specific 3D locations on the object that are constrained to remain fixed during the tracking optimization process. They act as stable reference points, ensuring that keypoints maintain consistent spatial relationships across different video frames and even different data collection sessions, which stabilizes the entire tracking solution.
Edge Length Constraints: This constraint forces neighboring keypoints to maintain a uniform distance between them throughout the movement. By enforcing this rule during the tracking calculation, the system prevents noisy fluctuations and sudden jumps in keypoint positions, resulting in smooth and temporally consistent trajectories suitable for physics simulations.
Point Cloud Differencing: This is an unsupervised method used to automatically separate the deformable object from its background in a point cloud. By comparing consecutive depth frames, the system filters out static background points and clusters the remaining points into a segmented object cloud, allowing autonomous detection even when objects are partially obscured.
Topology Identification: This step involves analyzing the local geometry of the segmented point cloud to determine which points belong to keypoints versus structural elements. This helps in identifying meaningful locations for tracking and ensures that the extracted data accurately reflects the underlying structure of the deformable object.

Terminology

Accuracy — 1× in this paper · explained in 1 episode
Benchmark — 1× in this paper · explained in 2 episodes
Consistency — 1× in this paper · explained in 3 episodes
Core — 1× in this paper · explained in 2 episodes
Downstream Tasks — 1× in this paper · explained in 2 episodes

Summary

The gist: TrackDeform3D presents an affordable and autonomous framework for collecting 3D datasets of deformable objects using only RGB-D cameras by identifying 3D keypoints and robustly tracking their trajectories, incorporating motion consistency constraints to produce temporally smooth and geometrically coherent data

Introduction

Deformable objects such as wire harnesses or fabrics are ubiquitous in homes and factories, yet remain challenging for robots to manipulate. Because these objects possess infinitely many degrees of freedom, directly modeling their full state is intractable. Researchers therefore often approximate their state with structured 3D representations such as keypoints or meshes, which provide a compact yet informative description of their geometry and topology. Large-scale, high-quality keypoint and mesh trajectory datasets of deformable objects could significantly advance robotic manipulation. Scalable data collection demands platforms that are easy to deploy, robust in operation, and require minimal human involvement. High data quality also demands two forms of geometric consistency: Initialization consistency requires that keypoints are placed at the same locations across different collection sessions, minimizing misalignment across data collection sessions. Trajectory consistency requires that distances between neighboring keypoints remain stable throughout a collection episode, preventing oscillating noise and sudden jumps that would corrupt downstream tasks such as computing stretching or bending forces in physics simulation.

Methodology

TrackDeform3D is a data collection pipeline for robust 3D keypoint tracking of deformable objects. The pipeline begins with an experimental setup where a single RGB-D camera is positioned in front of the data collection station, and two robotic arms grasp and manipulate the deformable object. The process involves several key stages:

Input: The system records a depth video sequence from time 0 to T, aiming to extract a sequence of 3D keypoints from the recorded video.
Point Cloud Segmentation Object Classification Anchor Points Detection Initialization Warmstart Topology Identification Initialized Keypoints Warmstart Anchor Points Detection Tracking Solve Filter Keypoints Final Frame? Yes No.
Problem Formulation: At each time step t, the system seeks to compute keypoint positions by minimizing an energy that encourages uniform spacing among neighboring keypoints until an equilibrium is reached. This optimization problem is formulated as arg min X(i,j)∈E xt,i − xt,j − dij squared s.t. xt,i ∈ Pseg t, xt,k = at k, ∀ k ∈ A. The anchor set A serves to constrain the solution space and stabilize the keypoint configuration across sessions.
Deformable Object Segmentation: This is achieved autonomously via point cloud differencing, where P1-P0 filters out static background points, and DBSCAN is applied to cluster the remaining points to yield the segmented point cloud Pseg 1.
Keypoint Initialization: This involves object classification based on local geometry eigenvalues and then determining anchor positions, warm-start positions using Farthest Point Sampling (FPS), and finally solving the constrained optimization problem to obtain the initial configuration X1.
Tracking: For subsequent frames t ≥ 2, anchor points are re-detected, and the keypoints are solved recursively using a GaussSeidel-style iterative projection procedure summarized in Algorithm 1, which is warm-started from Xt−1. A temporal moving average filter is then applied to suppress high-frequency jitter, yielding smooth and temporally consistent trajectories.

Experiments and Results

The experiments evaluate TrackDeform3D against several state-of-the-art tracking methods across diverse object categories and demonstrate consistent improvements in both geometric and tracking accuracy. The hardware setup includes two Franka Emika Panda robotic arms and an RGB-D camera capturing synchronized color and depth streams at 30 Hz. The dataset collection involved varying stiffness, appearance, and contact configurations for DLOs (rope/wire), BDLOs (branched rope/wire harnesses), and 2D objects (rectangular fabric/T-shirt). TrackDeform3D's performance is quantified using metrics such as Absolute edge length error (Abs.), reported as RMSE in millimeters, and Chamfer distance (CD). Table I shows that TrackDeform3D achieves lower Abs. and CD compared to baselines like CDCPD2 and SpatialTrackerV2. Ablation studies confirm the necessity of both anchor and edge length constraints during tracking, showing that removing either component causes keypoints to drift or lose consistent spacing.

Dataset Contribution

Using this framework, this paper presents a high-quality, large-scale dataset consisting of 6 deformable objects, totaling 110 minutes of trajectory data. This dataset addresses the scale and diversity gaps left by prior work, including the inclusion of branched deformable objects which are important for applications such as wire harness modeling. The dataset is annotated with 3D keypoint trajectories and topology information, and robot arm motion inputs will be accompanied by the indices of the grasped keypoints on the deformable object. This work provides a resource for training and benchmarking deformable object models.

Conclusion

TrackDeform3D successfully enforces anchor, edge length, and point cloud projection constraints to produce consistent and geometrically accurate keypoint trajectories across diverse object categories including DLOs, BDLOs, and cloth. The proposed pipeline yields a large-scale RGB-D dataset annotated with 3D keypoint trajectories and topology information, which is publicly released to support future research. Experimental results demonstrate that TrackDeform3D outperforms existing baselines across all evaluated metrics and object categories. The work confirms the contribution of each optimization component through ablation studies, validating the necessity of anchor and edge length constraints for accurate and stable tracking. Future work plans to address limitations such as the assumption of rectangular fabric geometry and unreliable anchor-point detection.

References

[1] J. Zhu, A. Cherubini, C. Dune, D. Navarro-Alarcon, F. Alambeigi, D. Berenson, F. Ficuciello, K. Harada, J. Kober, X. Li, J. Pan, W.

Improvements for AI systems

Robust initialization of deformable object keypoints across sessions can be achieved by enforcing geometric consistency through anchor points that are fixed during optimization, constraining the solution space and stabilizing the keypoint configuration across sessions. This enables a more reliable starting point for tracking than methods that rely on manual keypoint selection, which is labor-intensive and introduces initialization inconsistency across sequences.
Real-time tracking of deformable objects can be improved by using a unified constrained optimization framework that simultaneously enforces edge length and point cloud consistency. This prevents the oscillating noise and sudden jumps that would otherwise corrupt downstream tasks such as computing stretching or bending forces in physics simulation.
Autonomous segmentation of deformable objects can be enhanced by replacing manual prompting with an unsupervised procedure based on point cloud differencing. This allows for the extraction of object point clouds even in unstructured environments where deformable objects often occupy only a small portion of the scene, making robust unsupervised segmentation challenging.
The resulting improved system can generate large-scale, high-quality 3D datasets comprising 6 objects and 110 minutes of trajectory data, providing a valuable resource for training and benchmarking deformable object models. This addresses the large-scale gaps left by prior work in the field.

Abstract

Structured 3D representations such as keypoints and meshes offer compact, expressive descriptions of deformable objects, jointly capturing geometric and topological information useful for downstream tasks such as dynamics modeling and motion planning. However, robustly extracting such representations remains challenging, as current perception methods struggle to handle complex deformations. Moreover, large-scale 3D data collection remains a bottleneck: existing approaches either require prohibitive data collection efforts, such as labor-intensive annotation or expensive motion capture setups, or rely on simplifying assumptions that break down in unstructured environments. As a result, large-scale 3D datasets and benchmarks for deformable objects remain scarce. To address these challenges, this paper presents an affordable and autonomous framework for collecting 3D datasets of deformable objects using only RGB-D cameras. The proposed method identifies 3D keypoints and robustly tracks their trajectories, incorporating motion consistency constraints to produce temporally smooth and geometrically coherent data. TrackDeform3D is evaluated against several state-of-the-art tracking methods across diverse object categories and demonstrates consistent improvements in both geometric and tracking accuracy. Using this framework, this paper presents a high-quality, large-scale dataset consisting of 6 deformable objects, totaling 110 minutes of trajectory data. Project page: https://roahmlab.github.io/trackDeform3D-core-tracking/