Trajectory-based Diffusion from One-shot Human Video for Generalizable Manipulation

arXiv:2610.21792 · cs.RO · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: Today's paper: "Trajectory-based Diffusion from One-shot Human Video for Generalizable Manipulation".

Dev: The gist The authors introduce TDV, a trajectory-based diffusion framework that leverages data generation from one-shot human videos to enable generalizable manipulation across different embodiments and visual domains.

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So we're looking at this paper now, "Trajectory-based Diffusion from One-shot Human Video for Generalizable Manipulation." It looks like they’re trying to fix the problem of needing tons of data when you want a robot to learn a new task just by watching someone do it once.

Dev: Yeah, it sounds like the core idea here is using human video demonstrations to generate data that lets the robot work across different body types and visual settings without needing new training for every single setup.

Taro: It’s interesting because most existing methods still rely on collecting a lot of labeled data, which makes scaling up really hard for real-world deployment.

Rosa: Exactly. This paper introduces TDV, which is this trajectory-based diffusion framework that uses those single human videos to create generalizable manipulation policies.

Dev: The summary says TDV focuses on extracting task-relevant object trajectories using plug-and-play sixD pose estimation techniques to ignore the irrelevant backgrounds and different viewpoints.

Taro: So, if I'm driving a robot in a messy garage, this framework should be able to figure out where the object is relative to me without getting tripped up by the clutter?

Rosa: Right. The paper outlines three stages: video parsing to get those motion trajectories, trajectory generation where they separate hand tasks from object tasks, and then closed-loop execution when it's actually running.

Dev: For the hand tasks, they use a hand pose estimation model to reconstruct the human hand trajectories and then remap those directly onto what a two-finger gripper can do.

Taro: And for the object tasks, they apply that plug-and-play sixD pose estimation model and augment the data by sampling diverse initial poses to make sure it doesn't just learn one specific starting position.

Rosa: They represent trajectories relative to the reference object frame, which means even if you turn your camera around, the trajectory itself is consistent because it’s tied to that object.

Dev: The core learning mechanism is a trajectory-based diffusion policy modeled as f(T target t:t+nT target t−h:t, l, s), predicting the future trajectory based on the history and some task description and signal.

Taro: So it’s not just looking at where the object is now, but how it moved recently to predict what it will do next? That seems like a significant step toward understanding motion planning.

Rosa: It uses a Denoising Diffusion Probabilistic Model, or DDPM, to approximate the conditional distribution of that future trajectory based on the history and task description and procedure signal.

Dev: The encoder maps that history trajectory into a latent space using a two-layer MLP, while the task description is tokenized using BERT and the procedure signal is represented as a learnable embedding.

Title and authors: Taro: So they’re fusing different types of information—the motion, the text description, and some learned signals—before feeding it into a transformer encoder with positional information.

Rosa: Then, a DiT-block acts as the decoder backbone where those fused features and timestep embeddings guide the prediction of the next part of the trajectory.

Dev: The loss function they use is modified to L = MSE(ε k, εθ(O t, T n0 + ε k, k)), which basically means they are minimizing how far their predicted trajectory is from the actual future one given all that context.

Taro: That modification seems important because it allows the model to condition its prediction on both the current state and the specific task it's trying to achieve, which should help with robustness.

Rosa: After learning, they have this switch policy that governs transitions between subtasks, deciding whether to increment the procedure signal based on history and task description.

Dev: When a subtask finishes, this policy outputs a binary signal to change the gripper state and move to the next step in the procedure control signal.

Taro: That sounds like they’re building in some form of autonomy for multi-procedure tasks, where it can decide when to switch gears on its own rather than just following a fixed script.

Rosa: To make this work in the real world, during deployment, they perform real-time pose estimation of both the moving entity and the reference objects using FoundationPose to enable closed-loop control at a rate of thirty Hertz <ref:2610.21792#pg3>.

Dev: That thirty Hertz loop rate is pretty fast for controlling something physical, so you need low latency for that switch policy to feel responsive <ref:2610.21792#pg3>.

Taro: What about when the world misbehaves? If the pose estimation gets noisy during deployment, how gracefully does this system recover or handle that uncertainty?

Rosa: The authors acknowledge a limitation here: they note that the approach relies heavily on accurate sixD pose estimation, which limits its applicability to objects with complex geometry or deformable structures that are difficult to track reliably.

Dev: That makes sense. If the underlying pose estimation drifts, the entire trajectory prediction pipeline could fall apart quickly, which is a real concern for safety in real-time control loops.

Taro: And they also mention that spatial generalization is affected by the size of the generated training set; if you only have one demonstration, it might not generalize well to completely different environments.

Rosa: So while this framework handles visual domain changes okay because of its trajectory representation, you still need a decent amount of diverse data to ensure it performs well across radically different setups.

Title and authors: Dev: It seems like the big win here is moving away from just predicting the next frame and instead learning a sequence—a whole path—which is what allows for better state transitions.

Taro: So, looking at this paper, what I see is a system that learns how to move things by understanding their motion history rather than just static snapshots of where things are.

Rosa: That’s right. It gives us a way to learn manipulation policies directly from human video that scales better and handles visual changes reasonably well.

Dev: We've seen how the trajectory-based diffusion framework works, and it shows that using object-centric representations based on these trajectories is a solid intermediate step for cross-embodiment learning.

Taro: It’s interesting to think about what this means for future systems, where we might not have access to perfect pre-labeled datasets anymore, but we still have human video available.

Rosa: That’s the direction they are pushing toward. We're seeing how trajectory representations serve as an effective way to generate data efficiently for spatial generalization and robustness across camera viewpoints.

Dev: It validates that trajectories can be a powerful modality-agnostic intermediate representation, which is crucial if you want to train a policy that works on different physical arms or grippers later.

Taro: I just think the reliance on those 6D pose estimations is the main practical hurdle for now when we think about deploying this outside of perfectly controlled lab environments <ref:2610.21792#pg1>.

Rosa: So, to wrap up, the trajectory-based diffusion from one-shot human video for generalizable manipulation paper shows that by focusing on motion trajectories, we can generate policies that are robust to visual domain changes.

Dev: It also points toward a system where the history of movement is key for predicting future actions and managing complex multi-step procedures through a switch policy.

Taro: The limitation is definitely the need for accurate pose estimation and the dependency on having some amount of training data to achieve strong spatial generalization across different setups.

Rosa: So, this paper shows that by using human video to generate trajectories, we can get manipulation policies that generalize better across different bodies and viewpoints than before.

Dev: It’s a solid framework for generating data efficiently when you are trying to move toward more autonomous, multi-step manipulation tasks in the real world.

Taro: We'll keep an eye on how these trajectory representations perform when we start testing them in those messier, less controlled environments where the pose estimation might be less certain.

The paper's summary: Rosa: So we talked about how they use human video to generate data for robot learning, and now we need to look at what this paper actually says about how they made that process work.

Dev: Yeah, their main point is that this trajectory-based diffusion framework, TDV, lets a robot learn complex manipulation skills just from watching someone do it once in a video.

Rosa: Exactly. They’ve figured out how to extract these specific object trajectories from the video using some plug-and-play pose estimation techniques so they aren't bothered by weird backgrounds or different camera angles.

Dev: The core idea is that these extracted trajectories become this kind of intermediate representation, which means it doesn't care about the visual domain at all—it’s independent of whether you’re in a bright lab or a messy garage.

Rosa: It's inherently modality-agnostic, and that lets them generate data efficiently for things like cross-embodiment generalization.

Dev: They break the learning process into three parts: parsing the video to get those motion paths, generating the trajectories by separating hand tasks from object tasks, and then a closed-loop execution part for when it’s actually running.

Rosa: For hand tasks, they reconstruct what the human's hand path looks like and map it onto how a two-finger gripper moves.

Dev: And for object tasks, they use that same plug-and-play pose estimation model to pull out the object's path and then augment it by sampling lots of different starting positions.

Rosa: They represent these paths relative to the reference object frame, which is smart because that way, the trajectory itself is consistent even if you change your viewpoint.

Dev: The learning engine is a trajectory-based diffusion policy, which predicts the future path based on the history of where things have been, conditioned on some task description and procedure signal.

Rosa: They use a Denoising Diffusion Probabilistic Model to approximate the probability distribution of that future trajectory given what’s already happened in the history.

Dev: The input to this model is complex; they encode the history trajectory into a latent space, tokenize the task description with BERT, and fuse all that information using a transformer encoder before feeding it into a decoder backbone.

Rosa: Then, they use that fused information along with timestep embeddings to predict the next segment of the trajectory in this diffusion process.

Dev: The loss function is adjusted so that it minimizes how far their predicted future trajectory is from the actual one given all those inputs.

Rosa: They also have this switch policy that decides when to move from one subtask to the next, using both the history and task description to make that decision autonomously.

Dev: And for deployment, they use real-time pose estimation of both the moving object and reference objects—FoundationPose—to keep that closed-loop control running at thirty Hertz.

Rosa: So what this means for us is that we can generate manipulation policies directly from single human videos, which dramatically speeds up data collection and allows the robot to perform tasks across different physical bodies without needing a brand new training session every time.

Dev: It proves that focusing on motion trajectories as an intermediate representation is a solid way to get robust data for generalization across different visual settings.

Rosa: But they do have a caveat, which is important for me on the trials side—the whole thing relies heavily on accurate sixD pose estimation, so if you're working with something really complex or deformable, that tracking might become unreliable.

Dev: And they also noted that spatial generalization isn't perfect; it still depends on how much diverse training data you feed the system to make sure it works well in completely new environments.

Rosa: So while this is a powerful way to get started, we’re not yet seeing a fully autonomous system that handles every single physical scenario without needing some kind of fine-tuning or more robust tracking.

Dev: It’s still very much about building reliable data generation pipelines first, which then feeds into the learning models.

Rosa: That leads us nicely into how this works when we look at the real-world testing—how does this system actually handle those unpredictable moments when the world misbehaves during deployment?

The paper's improvements: Rosa: So we’ve looked at how TDV extracts those motion trajectories from videos to build a diffusion model for manipulation, and now we need to talk about what they suggest is better moving forward with this idea.

Dev: The authors point out a few things they think could really improve the system’s capability beyond what it does right now.

Rosa: One big suggestion is that instead of just predicting the next pose based on history, the trajectory-based diffusion policy should be designed to allow for revisiting previous poses, which is super important for tasks that have long sequences or complex steps.

Dev: Yeah, they suggest modifying the loss function in a way that conditions it more heavily on both the current state and what happened before—like L = MSE(epsilon k, epsilon theta(O t, T n0 + epsilon k, k)).

Rosa: That modification means the model gets a better signal to learn those long-term dependencies rather than just focusing on one step at a time.

Dev: It also emphasizes that the object-centric representation should be used to allow for simple trajectory scaling and interpolation, which helps mitigate the impact of background noise or different viewpoints during data augmentation.

Rosa: They are saying that by using this trajectory structure, we can create more robust data generation simply by manipulating the existing paths in a predictable way.

Dev: And for deployment, they stress that the switch policy should be trained on subtask segments so it becomes better at deciding when to transition between tasks automatically based on the history and description.

Rosa: That means moving toward a system that is more autonomous in its procedure, not just following a rigid script we hand-coded.

Dev: And for closed-loop control, they say we need to ensure that the real-time pose estimation during deployment happens at a fast rate, like thirty Hertz, for smooth physical interaction.

Rosa: So what this means is that the future work should focus on making those data augmentation techniques even more sophisticated and ensuring the switch policy handles unexpected failures or deviations in a more graceful way.

Dev: It seems like they are looking at how to make the system not just succeed in ideal conditions, but how it actually recovers when things get messy during real-time operation.

Rosa: That brings us right back to my question: if the underlying 6D pose estimation starts getting noisy because the object is hard to track, what’s the safety case for this entire closed-loop system <ref:2610.21792#pg1>?

Conclusion: Rosa: So we’ve gone through how they build TDV, from extracting those motion paths to using diffusion to predict future actions, and now we’re wrapping up this look at "Trajectory-based Diffusion from One-shot Human Video for Generalizable Manipulation."

Dev: Basically, the big picture is that by focusing on the trajectory rather than just static images of poses, they get a manipulation policy that works across different bodies and visual settings without needing massive amounts of new training data.

Rosa: It validates that human video can be a really efficient way to generate data for spatial generalization because those trajectories are inherently independent of the visual domain.

Dev: It also confirms that this trajectory representation is a solid intermediate step for cross-embodiment learning, which means we can train something once and have it work with different robot arms later on.

Rosa: But we have to remember the caveat: if you’re working with something really complex or deformable, the accuracy of those 6D pose estimations might limit how well this works in the real world <ref:2610.21792#pg1>.

Dev: And they also flagged that spatial generalization isn't perfect; it still depends on having a decent amount of diverse training data to make sure it performs well across radically different setups.

Rosa: So what this means for us is that we’ve got a framework that scales data generation using single demonstrations, but we still need to push the limits on tracking accuracy and data diversity for truly robust deployment.

Dev: I think the real change here is seeing how history—the past motion—is key for predicting future actions in complex multi-step procedures, which is what the switch policy manages.

Rosa: Exactly. It’s not just about what the object looks like right now, but where it has been and what the task requires next.

Dev: Moving forward, we need to see how this framework handles those real-time failure modes when things go wrong during actual robot operation, because that’s where we need more robustness.

Rosa: Right. And I'm curious about how this compares to other methods we've seen in the lab right now that focus on handling those unpredictable moments when the world misbehaves during deployment.

Guoping Pan, Xinmiao Yu, Wannan Cao, Jincheng Li, Baolin Jiang, Yizhe Li, Yi Cheng, ☘Houde Liu

Tsinghua University of Science and Technology of China

cs.RO

Submitted: 2026-10-15

Updated: 2026-10-15

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 87/100

The gist: The gist The authors introduce TDV, a trajectory-based diffusion framework that leverages data generation from one-shot human videos to enable generalizable manipulation across different embodiments

Key concepts

Trajectory-based Diffusion Framework (TDV): This is the core learning method where a diffusion model predicts future object trajectories based on past ones, conditioned on task descriptions. It uses a Denoising Diffusion Probabilistic Model (DDPM) to generate these sequences, effectively turning video motion into learnable data for robot control.
Modality-Agnostic Intermediate Representation: Instead of relying on raw visual pixels, the method converts human videos into sequences of 6D object poses. This trajectory format is independent of the specific camera used or the robot's physical body, making it a universal language that enables efficient data generation across different visual domains.
Plug-and-Play 6D Pose Estimation: This technique is used to extract object trajectories from videos without needing custom models for every object. It allows the system to quickly estimate the 3D location and orientation of objects in a video, ensuring that only task-relevant information is extracted, ignoring irrelevant backgrounds.

Summary

The gist The authors introduce TDV, a trajectory-based diffusion framework that leverages data generation from one-shot human videos to enable generalizable manipulation across different embodiments and visual domains.

How it works

TDV focuses on extracting task-relevant object trajectories from human videos using plug-and-play 6D pose estimation techniques to mitigate the impact of irrelevant backgrounds and varying viewpoints. These extracted trajectories serve as a modality-agnostic intermediate representation, which is inherently independent of the visual domains and different embodiments, thereby enabling efficient data generation. The framework is structured into three stages: video parsing to extract motion trajectories relevant to the manipulation task, trajectory generation where subtask segments are categorized into hand tasks and object tasks, and closed-loop execution during deployment.

For hand tasks, the method employs a hand pose estimation model to reconstruct human hand trajectories, which are then remapped to the two-finger gripper configuration. For object tasks, it adopts a plug-and-play 6D pose estimation model to extract object trajectories and performs data augmentation by sampling diverse initial poses. To eliminate the influence of viewpoint changes, trajectories are represented relative to the reference object frame. This results in a trajectory composed of a sequence of 6D object poses that encapsulates state transitions during manipulation.

Trajectory-based Diffusion Transformer

The core learning mechanism is a trajectory-based diffusion policy modeled as f(T target t:t+nT target t−h:t, l, s), where the future trajectory T target t:t+n of length n is predicted based on the history trajectory T target t−h:t of length h of the target object. This prediction is conditioned on the task description l and the task procedure signal s. A Denoising Diffusion Probabilistic Model (DDPM) is employed to approximate the conditional distribution p(T target t:t+nO t), where observations Ot are composed of the history trajectory T target t−h:t, task description l, and procedure signal s. The encoder maps the history trajectory to a latent space via a two-layer MLP. The task description l is tokenized using BERT, while the procedure signal s is represented as a learnable embedding. These features are then encoded with positional information according to their modalities and fused through a transformer encoder. A DiT-block serves as the decoder backbone, with the fused features and timestep embedding serving as the conditional information. The loss function is modified as L = MSE(ε k, εθ(O t, T n0 + ε k, k)).

Close-loop Execution

To achieve task autonomy for multi-procedure manipulation tasks, a switch policy governs transitions between subtasks. This policy determines whether to increment the procedure signal based on the history trajectory and task description. When a subtask is completed, the switch policy outputs a binary signal used to increment the procedure control signal and change the state of the gripper. The switch policy is trained using subtask segments, where each segment's final frame is identified as the transition point between states. During deployment, real-time pose estimation of both the moving entity and the reference objects is performed to facilitate closed-loop control.

Contributions

The paper makes three primary contributions:

Proposing a trajectory-based diffusion framework (TDV) that learns manipulation policies solely from human videos, significantly improving data collection efficiency and enabling scalable policy learning across diverse tasks.
Leveraging an object-centric intermediate representation based on trajectories to enable effective learning and cross-embodiment generalization.
Demonstrating that TDV achieves strong spatial generalization and remains robust under complex backgrounds and diverse camera viewpoints.

Experiments

The method was evaluated on RLBench simulation tasks, where the success rates for TDV achieved an average of 91.8% across all tasks, ranking first in 8 out of 12 tasks and achieving a 13.4% improvement over the previous state-of-the-art. In real-world experiments using a UR5 arm and RealSense D435 camera, TDV demonstrated success rates under varying demonstration counts, showing improvements as the number of training samples increased from one to fifty trajectories. Furthermore, spatial generalization was shown to improve significantly when expanding the dataset to 50 trajectories compared to a single demonstration, approaching 100% success rate. Visual domain generalization was confirmed by showing robustness across three camera viewpoints and four types of background perturbations.

Limitations

The authors note that the approach relies heavily on accurate 6D pose estimation, which limits its applicability to objects with complex geometry or deformable structures that are difficult to track reliably. The method's spatial generalization is also affected by the size of the generated training set.

The paper concludes that TDV enables generalizable manipulation across different embodiments from one-shot human demonstrations through efficient trajectory generation and a modality-agnostic intermediate representation. The framework successfully validates that trajectories serve as an effective intermediate representation, allowing efficient data generation for spatial generalization. The modality-agnostic nature of trajectories contributes to strong robustness across visual domains, enabling consistent performance under varying camera viewpoints and background conditions. However, the approach relies heavily on accurate 6D pose estimation, which limits its applicability to objects with complex geometry or deformable structures that are difficult to track reliably. The size of this generated training set affects spatial generalization.

The paper is supported by the Shenzhen Science and Technology Program under Grant RCJC20210706091946001 and Grant ZDCY2025090110420708. The work was also supported by Zerith Robotics.

The references cited include [8] H. Bharadhwaj et al., “Gen2act: Human video generation in novel scenarios enables generalizable robot manipulation,” arXiv preprint arXiv:2409.16283, 2024, [15] G. Pavlakos et al., “Reconstructing hands in 3d with transformers,” in CVPR, 2024, and [16] B. Wen et al., “Foundationpose: Unified 6d pose estimation and tracking of novel objects,” in CVPR, 2024. The references also include [35] J. Ho et al., “Denoising diffusion probabilistic models,” NeurIPS, 2020 and [36] J. Devlin et al., “Bert: Pre-training of deep bidirectional transformers for language understanding,” in NAACL, 2019. The references also include [37] S. James et al.

Improvements for AI systems

The TDV framework enables generalizable manipulation across different embodiments from oneshot human demonstrations, meaning a robot trained on one human video can perform similar tasks with different physical arms or grippers without retraining, as demonstrated by its success in crossembodiment generalization.
The trajectory-based diffusion policy, conditioned on the history trajectory rather than just current poses, allows for revisiting the same poses, which is crucial for tasks requiring complex sequences and enables better learning through the modified loss function: L = MSE(ε k, εθ(Ot, Tn 0 + ε k, k)) (5).
The framework utilizes an object-centric intermediate representation where trajectories are inherently independent of the visual domains and different embodiments, allowing for efficient data augmentation through simple trajectory scaling and interpolation, which mitigates the impact of irrelevant background and different viewpoints.
The switch policy, trained using subtask segments, governs transitions between subtasks to achieve task autonomy by determining whether to increment the procedure signal based on the history trajectory and task description.
Real-time execution is achieved by performing real-time pose estimation of both the moving entity and the reference objects during deployment using FoundationPose, enabling closed-loop control for multi-procedure manipulation tasks at a speed of 30 Hz.