Learning Where it Matters: Autonomous Data Collection and Offline Reinforcement Learning for Local Contact-Rich Manipulation

arXiv:2607.15982 · cs.RO · Submitted 2026-07-17 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Hi, I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "Learning Where it Matters".

Dev: Here's the gist of the paper. The gist: Learned policies trained end-to-end on large datasets often remain brittle in high-precision tasks and struggle with generalization, which this work addresses by proposing an automated data-collection scheme in combination with offline deep reinforcement learning for the critical segment of contact-rich tasks Key Insight and Problem Statement The limitations of end-to-end robot learning stem from a lack of structure and focus in data collection The authors argue that at least part of this gap is not merely a matter of scale, but of what data is collected and how tasks are structured Specifically, they show that high-precision tasks usually fail at the contact-rich, critical segment—such as an insertion with tight clearance They propose to exploit this structure explicitly by densely collecting data where it matters—at the critical segment Autonomous Data Collection Scheme The proposed scheme involves combining dense data collection with offline deep reinforcement learning (DRL) for policy learning This scheme eliminates the constant dependency on a human teleoperator The data collection process begins by replaying a single demonstration until the critical segment of the task is reached The exploration policy used for dense data collection is defined as πexplore(st) = at = a ∼ U(A) with probability p a∗t with probability 1 − p The greedy action is defined as a∗t = v · (sg + ∆sgrasp) − st∥(sg + ∆sgrasp) − st∥2 The task reward rt is a sparse reward determined by the distance of state st to the final state of the hardcoded trajectory sg: rt = Rsuccess · 1(∥st − (sg + ∆sgrasp)∥2 ≤ ϵ Policy Learning and Deployment The learned policy πθ is obtained by combining data collection with offline DRL The learning algorithm builds on top of SAC with improvements for sample efficiency Interaction during critical interactions employs hybrid force-position control The central claim is that traditional planning combined with data collection and learning for the critical segment leads to high success rates and robust policies The method performs well in out-of-distribution scenarios where end-to-end approaches struggle A key finding is that the sequential pipeline achieves consistently ≥ 90% SR Ablation and Robustness The ablation study on data collection parameters shows that successful policies require dense data coverage of the critical segment Policies trained on little exploration (p <= 0.3) achieve low success rates Multi-modal sensing for high success rates shows that combining a wrist-mounted F/T sensor with a single wrist camera achieves 100 % success in simulation The method maintains high success rates even under OOD scenarios, unlike end-to-end methods By adopting a modular approach with targeted data collection, the paper concludes that focusing collection on tasks and task-segments that are otherwise difficult to solve is key to robust generalizable manipulation The paper demonstrates that targeted reliance on robot data and structured task decomposition are key enablers of robust generalizable manipulation Limitations The paper notes limitations regarding task diversity, stating that sequential boundaries are harder to define for tasks like t-shirt folding Hardware precision is also a limitation, as the method benefits from hardware that precisely reaches target poses for waypoint tracking and policy switching<ref:2607.

Paper summary: Rosa: So we just covered the main idea of this paper, which is that traditional end-to-end learning struggles with precision tasks because they lack focus in data collection. The authors argue the solution is to leverage dense data collection only for the critical segment of contact-rich tasks.

Dev: Exactly, and they show how combining that collected data with offline deep reinforcement learning creates a policy pi theta that performs well when sequentially chained with off-the-shelf pose estimation and motion planning.

Taro: I like the idea of replaying a single demonstration until the critical segment is reached before starting dense data collection using a greedy-random exploration policy. That seems like a structured way to focus the data gathering effort.

Rosa: It’s about eliminating that constant dependency on a human teleoperator's skill, and also removing the need for online policy updates during deployment, which simplifies things a lot for real-time control.

Dev: And they model contact-rich interaction as an MDP, but they focus their learning goal only on maximizing the expected discounted return J(pi theta) = E

P infinity t=zero gamma t r t: within that critical segment <ref:2607.15982#pg3,model contact-rich interaction as an MDP>.

Rosa: They’re not trying to solve the whole task with one massive model; they are exploiting the structure of contact-rich manipulation itself by learning only what is necessary for robustness.

Taro: So, if you look at how they set up the autonomous offline data collection, it seems like they first record a single demonstration using kinesthetic teaching before replaying it until that critical segment is hit.

Dev: Right, and then during that replay phase, they execute a mixed greedy-random policy pi explore to gather both successful and failed rollouts for the critical part of the task.

Rosa: And then they use all that collected data to obtain the learned policy pi theta through offline DRL training, which is much more sample efficient than trying to train it end-to-end from scratch.

Taro: The key seems to be that this sequential pipeline—data collection, learning the critical segment, and then chaining it with off-the-shelf planning—leads to consistently high success rates above ninety percent.

Dev: It changes how we think about these complex manipulation tasks; instead of a single monolithic policy, we can focus on targeted data collection for the hardest parts.

Conclusion: Rosa: So to wrap up this discussion on "Learning Where it Matters: Autonomous Data Collection and Offline Reinforcement Learning for Local Contact-Rich Manipulation," the main point is that focusing data collection only on the critical segment of contact tasks yields high success rates.

Dev: The paper by Hausdorfer et al. shows that this modular approach, combining autonomous data collection with offline deep reinforcement learning, is a viable way to achieve robust manipulation policies.

Taro: It suggests that for precision applications, we can move away from training massive end-to-end models and instead use targeted reliance on robot data and structured task decomposition.

Rosa: The implication for the world is that this method opens the door for targeted data collection in contact-rich tasks, leading to high success rates in precision applications across many different scenarios.

Dev: It means we can tackle these challenging physical interactions with policies that are more reliable and better at handling unexpected situations than those trained end-to-end.

Oliver Hausdorfer, Linus Schwarz, Gabor Marko, Christian Dietz, Timo Class, Luka Hofer, Jim Yun-Jin Li, Johannes Hechtl Ralf Romer Gabor Marko Angela P. Schoellig

TU Munich University of Munich and Siemens AG

cs.RO

Submitted: 2026-07-17

Updated: 2026-10-20

Comments: Project webpage: https://anonymous.4open.science/w/data_and_learning_where_it_matters/

Code: https://github.com/huggingface/lerobot

Project page: https://anonymous.4open.science/w/data_and_learning_where_it_matters

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: The gist: Learned policies trained end-to-end on large datasets often remain brittle in high-precision tasks and struggle with generalization, which this work addresses by proposing an automated

Key concepts

End-to-End Learning Limitations: Policies trained directly from raw data often lack the structure needed for precision tasks. They struggle because the required data is too scarce or poorly targeted, leading to brittle performance when faced with real-world variations outside the training environment.
Autonomous Data Collection Scheme: This scheme automatically collects dense sensor data only in the difficult, contact-rich parts of a task. It replays demonstrations until it hits these critical segments, ensuring high-quality data is gathered locally without needing a human operator for every interaction.
Offline Reinforcement Learning (Offline RL): This technique uses pre-collected, static datasets to train policies. The collected contact data is used to create robust local policies that can be deployed alongside traditional motion planning, allowing the system to handle complex interactions effectively.
Modular Approach: The method combines three components: targeted autonomous data collection for critical segments, offline RL for policy learning in those segments, and classical motion planning for the overall task structure. This modularity allows the system to leverage the strengths of each method.

Terminology

Reinforcement Learning — 2× in this paper · explained in 4 episodes
Deep Reinforcement Learning — 1× in this paper · explained in 3 episodes
Generalization — 1× in this paper · explained in 13 episodes

Summary

The gist: Learned policies trained end-to-end on large datasets often remain brittle in high-precision tasks and struggle with generalization, which this work addresses by proposing an automated data-collection scheme in combination with offline deep reinforcement learning for the critical segment of contact-rich tasks

Key Insight and Problem Statement

The limitations of end-to-end robot learning stem from a lack of structure and focus in data collection The authors argue that at least part of this gap is not merely a matter of scale, but of what data is collected and how tasks are structured Specifically, they show that high-precision tasks usually fail at the contact-rich, critical segment—such as an insertion with tight clearance They propose to exploit this structure explicitly by densely collecting data where it matters—at the critical segment

Autonomous Data Collection Scheme

The proposed scheme involves combining dense data collection with offline deep reinforcement learning (DRL) for policy learning This scheme eliminates the constant dependency on a human teleoperator The data collection process begins by replaying a single demonstration until the critical segment of the task is reached The exploration policy used for dense data collection is defined as πexplore(st) = at = a ∼ U(A) with probability p a∗t with probability 1 − p The greedy action is defined as a∗t = v · (sg + ∆sgrasp) − st∥(sg + ∆sgrasp) − st∥2 The task reward rt is a sparse reward determined by the distance of state st to the final state of the hardcoded trajectory sg: rt = Rsuccess · 1(∥st − (sg + ∆sgrasp)∥2 ≤ ϵ

Policy Learning and Deployment

The learned policy πθ is obtained by combining data collection with offline DRL The learning algorithm builds on top of SAC with improvements for sample efficiency Interaction during critical interactions employs hybrid force-position control The central claim is that traditional planning combined with data collection and learning for the critical segment leads to high success rates and robust policies The method performs well in out-of-distribution scenarios where end-to-end approaches struggle A key finding is that the sequential pipeline achieves consistently ≥ 90% SR

Ablation and Robustness

The ablation study on data collection parameters shows that successful policies require dense data coverage of the critical segment Policies trained on little exploration (p <= 0.3) achieve low success rates Multi-modal sensing for high success rates shows that combining a wrist-mounted F/T sensor with a single wrist camera achieves 100 % success in simulation The method maintains high success rates even under OOD scenarios, unlike end-to-end methods

Conclusion

By adopting a modular approach with targeted data collection, the paper concludes that focusing collection on tasks and task-segments that are otherwise difficult to solve is key to robust generalizable manipulation The paper demonstrates that targeted reliance on robot data and structured task decomposition are key enablers of robust generalizable manipulation

Limitations

The paper notes limitations regarding task diversity, stating that sequential boundaries are harder to define for tasks like t-shirt folding Hardware precision is also a limitation, as the method benefits from hardware that precisely reaches target poses for waypoint tracking and policy switching<ref:2607.

Improvements for AI systems

Data Collection Scheme for Contact-Rich Tasks: Implement an automated data acquisition scheme that performs dense data collection at critical segments using mixed greedy-random exploration, as described in Section 3.2. This eliminates reliance on a human teleoperator and provides dense coverage of the critical segment while maintaining high success rates.
Policy Learning with Sample Efficiency: Train policies using an offline deep reinforcement learning approach based on SAC, employing improvements like layer normalization and Randomized Ensembled Double Q-Learning to mitigate Q-function overestimation, ensuring sample-efficient learning algorithm for minimizing data collection effort.
Deployment Strategy with Sequential Policies: Integrate the learned policy with traditional planning for free-space motion and switch policies based on contact events and Q-function values, as proposed in Section 3.4. This allows the robot to execute sequential policies that are tailored to different task segments, achieving success rates above 94% across all tasks.
Hybrid Control for Critical Interactions: Employ hybrid force-position control during critical, force-sensitive interactions like insertion, applying a desired force along the insertion axis while position controlling other degrees of freedom, as detailed in Section 3.2 and Appendix A.3 to enable compliant and force-controlled behavior with precise positioning.
Success Classification via Q-function: Utilize the learned Q-function as a success classifier during deployment by setting a threshold lambda success, as shown in Equation (4) and refined with hysteresis detection (Equation 8), which allows for "success = 1 if Q(st, πθ(st)) > λsuccess or H(t) 0 otherwise."
Enhanced Pose Estimation Pipeline: Integrate language-guided image segmentation models and off-the-shelf pose estimation models sequentially to obtain 6D pose estimates for manipulation, as outlined in Section 3.4. This pipeline allows the robot to handle language-guided segmentation model (SAM3 [8]) and off-the-shelf pose estimation model [9, 45] for real-world object interaction.
Robustness to Out-of-Distribution Scenarios: Design policies that rely on local camera images and utilize sequential policies to leverage task segment specific priors, which increases performance significantly for difficult tasks and maintains high success rates in OOD scenarios where end-to-end approaches struggle.

Abstract

Learned policies trained end-to-end on large datasets often remain brittle in high-precision contact-rich tasks. Further, they typically require large amounts of manually collected data and still struggle to generalize to scenes outside of the training scenarios. Classical motion planning, in turn, generalizes well but lacks the precision and contact awareness for tight-tolerance tasks. To reliably solve challenging contact-rich tasks, we propose to leverage robot data and learning where it matters in a modular approach. First, we introduce an autonomous data collection scheme for the real world that collects dense data locally in the contact-rich segment of the task, eliminating reliance on a teleoperator's skill. Second, we show that using offline RL, the collected data can be translated into policies with high success rates. We then integrate our local policies with traditional motion planning to solve long-horizon assembly tasks. Across four real-world tasks, using only 2-2.5 h of autonomous data collection, we achieve an average success rate of 96% for long-horizon tasks, compared to 55% for a planning-only baseline and 37% for the strongest end-to-end baseline. Our local policies also reduce applied forces by 49%. Notably, performance remains at 90% for scene-level out-of-distribution scenarios, while end-to-end approaches struggle. In total, we run over 1500 real-world evaluation rollouts. Our results pave the way for targeted data collection in contact-rich tasks and for high success rates in precision applications. Videos and qualitative results are available at https://learnsyslab.github.io/data and learning where it matters/