Spatially Conditioned Diffusion Policy: Learning Precise and Robust Manipulation with a Single RGB Camera

arXiv:2606.14535 · cs.RO · Submitted 2026-06-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Spatially Conditioned Diffusion Policy".

Dev: The gist:

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: So we're looking at this paper called Spatially Conditioned Diffusion Policy: Learning Precise and Robust Manipulation with a Single RGB Camera. Basically it tackles the problem that most visual imitation learning uses multiple cameras with wrist-mounted views because that gives you good local detail, but doing it from just one global view is really hard for the policy to figure out where to look.

Dev: Right. The paper proposes this Spatially Conditioned Diffusion Policy, or SCDP, and their core idea is that end-effector trajectories can act as visual attention anchors that point towards the task-relevant regions in a single image.

Taro: It claims they can achieve precise and robust manipulation even with only a single RGB camera, which is pretty impressive when you think about how much detail you need for things like USB insertion or grasping objects.

Rosa: Exactly. The thesis is that by using those intermediate end-effector trajectories to sample point-wise features from multi-scale image feature maps, the policy can extract those fine visual details while still handling the broader context of what's happening in the scene.

Dev: They built this system with two main parts: first, a visual encoder that generates multi-scale feature maps to get both broad context and fine details, and second, a spatial conditioning module that samples features along those intermediate trajectories during the diffusion process.

Taro: I’m curious about how they actually do that sampling part. Because the whole point hinges on using those trajectories as anchors for visual attention.

Rosa: That's where it gets interesting. In this paper, they reconstruct future trajectories from an intermediate action sample at a given denoising step, project those three dee points onto the image plane using camera intrinsics, and then sample point-wise features from all those multi-scale feature maps <ref:2606.14535#pg2>.

Dev: So they aggregate all those sampled point-wise feature vectors into one spatial context vector that conditions the diffusion U-Net along with the robot's state to denoise the action samples. That’s how they feed it into the process, according to their Diffusion Policy framework.

Taro: It sounds like they are essentially guiding the AI on where to pay attention in real time by mapping its planned movement back onto a visual feature map. But what about when things go wrong? How does it handle visual distractors?

Rosa: The results show that SCDP consistently outperforms strong single-view baselines and actually achieves performance comparable to those setups that use multiple cameras, even when only one camera is present.

Dev: They tested this in a simulation setting where they found SCDP had over eighty percent success in every Meta-World difficulty group using only twenty demonstrations and a single global camera.

Taro: That's a lot of data efficiency for imitation learning, especially for complex tasks like the ones they tested. What about when we move it outside the lab? Rosa, what does that look like in practice?

Rosa: In real-world experiments, SCDP maintained high accuracy on precise tasks and showed robustness when there were visual distractors around the robot. They specifically found that focusing on regions along the predicted end-effector trajectory reduces sensitivity to surrounding clutter.

Paper summary: Dev: That’s a key finding for me because it means we don't need that extra camera setup for every single application if the policy can learn to anchor itself visually this way, right?

Taro: It suggests that grounding visual conditioning along the estimated end-effector trajectory allows the system to focus on what matters while staying robust against environmental noise.

Rosa: Exactly. The ablation study also showed that using all feature levels from one through five provided peak performance of eighty-two point five percent, which was a improvement over just using the finest level, F5.

Dev: And they noted that the choice of visual backbone matters too; standard ResNets did a better job preserving locality bias across different resolutions compared to models like ViT-S/sixteen or CLIP-pretrained ResNet-fifty.

Taro: So it seems the architecture is quite flexible, but the underlying mechanism is still heavily reliant on that trajectory reconstruction and spatial conditioning step.

Rosa: And they also found that increasing the reconstruction horizon s consistently improved success rates, meaning incorporating future trajectory points gave it richer context for action generation.

Dev: It’s interesting because it tells us something about what context a policy needs to make a good decision; it’s not just the immediate state, but where it intends to move next.

Taro: I have to say though, the authors did flag some limitations in their conclusion. They pointed out that the method assumes task-relevant visual information lies near the end-effector trajectory, which might limit its usefulness for tasks that require indirect interaction or tool use further away from the hand.

Rosa: That’s a fair limitation. It works really well for contact-rich manipulation like insertion and grasping where the end effector is key, but if you're working with something more abstract, it might not generalize as widely.

Dev: And another thing they mentioned is that extracting reliable visual features from the attention region can be really challenging when things are heavily occluded. That’s a real hurdle for any vision-based system anyway.

Taro: So for anyone listening who is thinking about applying this, the main thing to keep in mind is that it’s optimized for tasks where the end effector path is a good visual cue, and you have to be careful with heavy occlusion.

Rosa: Right. So we’ve covered what this Spatially Conditioned Diffusion Policy does, how it uses trajectory anchors on multi-scale features to get precise control from one camera, and the practical results showing it beats multi-camera setups in many ways.

Dev: And for the conclusion of this talk, the paper calls their work Spatially Conditioned Diffusion Policy: Learning Precise and Robust Manipulation with a Single RGB Camera.

Taro: It really demonstrates how visual conditioning along an estimated end-effector trajectory enables the system to focus on task-relevant regions while remaining robust to surrounding clutter.

Rosa: That’s what it boils down to, showing the potential of single-camera imitation learning for high precision in manipulation tasks.

Conclusion: Rosa: So we're wrapping up this discussion on Spatially Conditioned Diffusion Policy by learning precise and robust manipulation with just one RGB camera.

Dev: The authors are focusing on using end-effector trajectories as visual attention anchors to get that precision in a single view setting.

Taro: It seems like the big idea is forcing the policy to look at specific parts of the image that matter for the next action step.

Rosa: Right, and it’s about making sure the AI doesn't just see everything equally, but focuses its vision where it needs to be for a given movement.

Dev: The implication here is that we might not always need those multi-camera setups if we can get this level of spatial conditioning right.

Taro: But there are still caveats, because the authors admit the method relies on task-relevant visual information being near the end effector trajectory, which limits it for things like indirect tool use.

Rosa: That’s true, and they also mentioned that getting reliable visual features when things are heavily occluded is really hard.

Dev: So we've seen how the math works and what the numbers say about performance against multi-camera baselines.

Taro: The real-world tests showed it was still competitive even with just one camera, which is a solid result for practical application.

Rosa: It opens up a lot of possibilities for deploying robots in environments where we only have access to one main camera.

Dev: It’s definitely interesting to see how diffusion models can be steered this way without needing extra sensor data during the actual execution loop.

Korea Advanced Institute of Science and Technology (KAIST) · Neuromeka

cs.RO

Submitted: 2026-06-12

Updated: 2026-10-15

Comments: Accepted at CoRL 2026

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: The gist: Spatially Conditioned Diffusion Policy (SCDP) achieves precise and robust manipulation in a single-camera setting by using intermediate end-effector trajectories as visual attention anchors

Key concepts

Multi-Scale Image Encoder
This component uses a ResNet backbone to generate several feature maps (F1 through F5) with different resolutions. These maps capture both broad scene context and very fine visual details simultaneously. This ensures the policy has access to information at various levels of detail, from general surroundings to specific interaction points.
Spatial Conditioning Module
This module uses intermediate end-effector trajectories as anchors to sample visual features from the multi-scale maps. By projecting these 3D trajectory points onto the 2D image plane, it identifies specific locations of interest. Features are then extracted at these anchor points and aggregated to create a spatial context vector for the policy.
Diffusion Policy (DP) Framework
SCDP is formulated as a conditional diffusion process where the denoising network is guided by this spatial context vector. Instead of just using the current state, it incorporates Fk, which represents task-relevant visual information sampled along the trajectory. This conditioning helps guide the policy to generate more accurate and robust actions.
End-Effector Trajectory as Attention Anchors
The core idea is that where the robot's tool or gripper is going (the end-effector) reflects what part of the scene is important for the current task. The system reconstructs this trajectory during denoising and uses these points to sample visual features, effectively telling the policy where to look visually to perform its next move.

Terminology

Summary

The gist: Spatially Conditioned Diffusion Policy (SCDP) achieves precise and robust manipulation in a single-camera setting by using intermediate end-effector trajectories as visual attention anchors to sample point-wise features from multi-scale image feature maps, allowing it to capture fine details and identify task-relevant regions

Spatially Conditioned Diffusion Policy (SCDP) Overview

The paper introduces SCDP, a diffusion-based visuomotor policy designed to enable precise and robust manipulation using only a single RGB camera. The core idea is that endeffector trajectories can serve as visual attention anchors that reflect task-relevant regions. SCDP consists of two key components: (i) a visual encoder that produces multi-scale feature maps to capture both broader context and fine-grained visual features, and (ii) a spatial conditioning module that samples point-wise features along intermediate end-effector trajectories in the diffusion loop. Extensive simulation experiments show that SCDP consistently outperforms strong single-view baselines and achieves performance comparable to multi-camera baselines.

Multi-Scale Image Encoder

SCDP leverages a set of multi-scale feature maps to capture both global scene context and finegrained visual details required for precise interaction. The architecture employs ResNet-18 as the backbone, where stages produce feature maps with effective strides of 4, 4, 8, 16, and 32 relative to the input resolution. Independent 3×3 convolutions are applied to each stage Ci to align channel dimensions across different feature maps to d and provide smoother local features for the subsequent denoising process. This results in a set of multi-scale feature maps F = F1,..., F5, where each Fi denotes the feature map of the corresponding stage Ci.

Spatial Conditioning Module

The spatial conditioning module utilizes intermediate action trajectories to sample point-wise visual features at task-relevant locations. This process involves three steps: (i) reconstructing the end-effector point trajectory from an intermediate action trajectory Ak t at denoising step k, (ii) projecting the reconstructed trajectory onto the image plane, and (iii) sampling point-wise features from the multi-scale feature maps F. The reconstruction involves parameterizing actions as a relative displacement from the current state to construct a point trajectory P0:s, where subsequent points Pi are iteratively updated throughout the denoising process. The 3D points in task space are then projected into 2D pixel coordinates (ui, vi) using the camera intrinsic matrix K to determine the locations of the attention anchors. Point-wise feature vectors fi,n are extracted from these anchors using bilinear interpolation and aggregated across multiple resolutions and trajectory points into a single spatial context vector F = [f¯1,..., f¯5].

Diffusion Process

The policy is formulated as a conditional diffusion process following the Diffusion Policy (DP) framework, where the denoising network is conditioned on the spatial context Fk instead of the 4th page. The training loss is defined as L = Ek,A0,ϵ∥ϵ − ϵθ(Ak,Fk, k)∥2. The action denoising network is built upon a conditional U-Net backbone where the spatial context Fk is incorporated via Feature-wise Linear Modulation (FiLM). During inference, the DDIM scheduler is employed to enable high-quality action generation with fewer denoising steps.

Experimental Validation and Results

The experiments validate SCDP through simulation and real-world evaluations across diverse manipulation tasks. SCDP achieves over 80% success in every Meta-World difficulty group using only 20 demonstrations and a single global camera. Experiment 2 demonstrates that spatial conditioning helps SCDP capture task-relevant visual information, as it outperforms baselines like DP, SKIL, and OTTER by a large margin. Furthermore, Experiment 3 shows that SCDP remains competitive even when using only a single RGB camera compared to baselines equipped with an additional wrist-view camera. Real-world results show SCDP maintains high accuracy on high-precision tasks and robustness in distractor-rich settings, suggesting that grounding visual conditioning along the estimated end-effector trajectory enables SCDP to focus on task-relevant regions while remaining robust to surrounding clutter.

Ablation Study Insights

The ablation study confirms the benefits of the proposed mechanism, showing that spatial conditioning provides a useful inductive bias toward task-relevant regions, enabling more data-efficient learning. Utilizing all feature levels (F1–F5) in the multi-scale configuration leads to peak performance reaching 82.5%, which is a 13.9%p improvement over the F5-only variant. The choice of visual backbone also impacts performance, with standard ResNets better preserving locality bias across resolutions than ViT-S/16 or CLIP-pretrained ResNet-50. The reconstruction horizon s significantly affects success rates, with performance improving consistently as the horizon increases, suggesting that incorporating future trajectory points provides richer task-relevant context for action generation.

Conclusion and Limitations

SCDP successfully demonstrates the potential of single-camera imitation learning by achieving performance comparable to a baseline equipped with an additional wristview camera. However, the paper notes that SCDP assumes that task-relevant visual information lies near the endeffector trajectory, which may limit its applicability to tool-use or indirect-interaction tasks. Additionally, extracting reliable visual features from the attention region becomes challenging in heavily occluded settings. Future work is suggested to extend the attention region beyond the endeffector trajectory and leverage complementary sensing modalities.

Real-World Task Performance

In real-world experiments, SCDP performs challenging tasks like USB Insertion and Battery Insertion by leveraging the reconstructed end-effector trajectory as attention anchors. For distractor robustness, SCDP focuses on regions along the predicted end-effector trajectory, which reduces sensitivity to surrounding clutter. The results show SCDP achieves high success rates on tasks requiring precise geometric alignment and robust grasping in unstructured environments.

Implementation Details

The implementation details specify training hyperparameters, including 1000 epochs for Meta-World and 3000 epochs for Real-World DexArt, with a reconstruction horizon of 8. The diffusion head utilizes U-Net dimensions (128, 256, 384) and the noise scheduler is DDIM. The observation horizon is set to 2 and the action horizon to 8. The multi-scale image encoder uses backbone channels (64, 64, 128, 256, 512) and an output dimension of 320 for the feature levels F1–F5.

Improvements for AI systems

  1. Improve precise manipulation in single-camera settings by implementing Spatially Conditioned Diffusion Policy (SCDP), which uses intermediate denoising trajectories as spatial attention anchors on multi-scale image feature maps, allowing the policy to extract task-relevant visual features while preserving fine visual details. This allows the system to achieve precise and robust manipulation in a single-camera setting, specifically in tasks like USB insertion.

  2. Enhance robustness to visual distractors by leveraging the trajectory information for spatial conditioning, as SCDP focuses on regions along the predicted end-effector trajectory, such as the cup handle and cup wall during approach, it reduces sensitivity to surrounding clutter. This enables effective manipulation in distractor-rich settings compared to baselines like DP.

  3. Achieve superior data efficiency by utilizing task-relevant visual features for learning, as SCDP exhibits superior data efficiency, supporting the effectiveness of its visual representations, leading to a higher performance ceiling when demonstrations increase compared to methods like DP and DP3.

  4. Improve generalization across manipulation settings by integrating multi-scale image encodings, which allows the system to capture both coarse scene context and fine-grained visual details required for precise interaction. This ensures the policy can handle diverse tasks, as shown by its high success rates across Meta-World and DexArt benchmarks.

  5. Increase competitive performance in single-camera regimes by demonstrating that SCDP achieves performance comparable to a baseline equipped with an additional wristview camera while relying solely on a single RGB camera. This validates the potential of single-camera imitation learning for complex tasks.

Abstract

Recent visual imitation learning systems have widely adopted multi-camera setups with wrist-mounted cameras as the de facto standard. However, manipulation from a single global view remains challenging, as the policy should capture fine-grained interaction details and identify task-relevant regions without local wrist views. To address this challenge, we present Spatially Conditioned Diffusion Policy (SCDP), a diffusion-based visuomotor policy that achieves precise and robust manipulation in a single-camera setting. Our key idea is that end-effector trajectories can serve as visual attention anchors that reflect task-relevant regions. Building on this idea, SCDP consists of two key components: (i) a visual encoder that produces multi-scale feature maps to capture both broader context and fine-grained visual features, and (ii) a spatial conditioning module that samples point-wise features along intermediate end-effector trajectories in the diffusion loop. Extensive simulation experiments show that SCDP consistently outperforms strong single-view baselines and achieves performance comparable to multi-camera baselines. Real-world experiments further demonstrate precise manipulation and robustness to visual distractors, highlighting the potential of single-camera imitation learning.

Sources

Related papers