MoCaP: Mobile Code-as-Policies for Zero-Shot Mobile Manipulation from Passive Human Videos

arXiv:2610.20998 · cs.RO · Submitted 2026-10-14 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "MoCaP: Mobile Code-as-Policies for Zero-Shot Mobile Manipulation from Passive Human Videos".

Dev: The gist The MoCaP framework turns passive human video into guidance for an end-to-end mobile manipulation policy by extracting spatial memory and cues of how people prepare to interact with…

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So we've talked about how MoCaP works and what those initial results look like, and now let's talk a bit more about the specific structure of this paper, focusing on what they call its improvements.

Dev: They focus on five main improvements in the MoCaP: first, they have a unified Code-as-Policy system that coordinates navigation and manipulation without needing to predefine those two stages separately.

Taro: That seems like a big step because it means the system can revise actions from both planning domains at the same time based on feedback from live geometry, which is crucial for real-time control.

Rosa: Right, so if it tries to move and bumps something, it can immediately adjust its plan for both where to go next and how to grab things simultaneously.

Dev: Second, they enhance object localization via this fine-grained spatial memory that extracts object relations, landmarks, and place connectivity from the videos. They claim this increases navigation success from seventy percent up to one hundred percent >

Taro: That improved localization means the robot can search much more effectively because it understands the context of where objects are in relation to each other across different places in those recordings.

Rosa: And third, they have human-guided base placement refinement, which uses that human approach prior to guide docking by inferring the direction from how a person positioned themselves before manipulation >

Dev: That's interesting because it turns arrivals into graspable configurations using only video data accumulated from everyday activity. It’s using behavior to solve a geometric problem.

Taro: It seems like they are turning the preparation phase of human interaction into an explicit guidance signal for the robot's approach, which is something that was missing in prior work >

Rosa: And fourth, they have closed-loop policy revision where the code generation policy is revised based on execution feedback; failed actions interrupt the remaining program to allow for plan revision >

Dev: That's a safety feature. If the robot messes up a step, it doesn't just keep going blindly; it stops and regens its plan in real-time against live constraints.

Taro: And fifth, they have task-independent memory reuse, meaning the knowledge extracted from passive videos is not tied to one specific request but serves as a persistent guidance layer for any open-vocabulary instruction >

Rosa: So essentially, they've built a reusable library of human interaction knowledge that the robot can tap into for many different tasks without needing a new demonstration every time.

Dev: That reusability is what makes it flexible for open-vocabulary systems; it’s not just solving one problem, but setting up the robot to solve many problems >

Taro: So the structure of MoCaP seems designed to handle the messy reality of real-world interaction by separating how we store that knowledge from how we execute the control policy >

The paper's summary: Rosa: Let's go back to what this whole process means in practice. The paper summarizes MoCaP as an agentic framework that takes passive human video recordings and turns them into guidance for an end-to-end mobile manipulation policy.

Dev: Essentially, the system extracts spatial memory and cues about how people prepare to interact with objects, which then informs both the navigation part—how it searches—and the search for a manipulation-ready configuration.

Taro: So instead of just having pre-programmed paths or fixed goal points, the robot gets dynamic guidance based on what it learns from human preparation in those videos >

Rosa: That's right, and they use this guidance together with current observations to coordinate navigation and manipulation primitives using a code agent.

Dev: This code agent generates Python programs that invoke robot-grounded primitives, like lift grasped object or prepare for manipulation, which are then executed in a closed loop.

Taro: The key is that the execution feedback from these primitives informs the next step of code generation, which lets it adjust its plan against live geometry and kinematic constraints >

Rosa: It’s this continuous loop of guidance from human video memory and real-time execution feedback that allows it to solve motions under the robot’s own physical limitations.

Dev: The system is designed to be robust because it relies on this multimodal input—the learned spatial knowledge from video and the current visual observations—to coordinate everything needed for a task.

Taro: It moves beyond just simple perception; it uses human activity as a source of complementary information about how interaction is prepared and performed, which motivates using human behavior as more than just a record of object locations >

The paper's improvements: Rosa: We talked about the specific mechanisms, but let's focus on what these improvements actually mean for how we build these systems.

Dev: The first improvement is that they don't separate navigation and manipulation stages in control; they use this unified Code-as-Policy system to coordinate them flexibly through executable code >

Taro: That means the planning domain can simultaneously revise where to go and what action to take based on feedback, which is much more efficient than having a strict handover between two different control modules >

Rosa: And the second improvement is that they developed this fine-grained spatial memory that draws on everyday human activity recordings to extract object relations, landmarks, and place connectivity for subsequent robot search and navigation >

Dev: That means the system gains much better object localization because it understands how objects relate to each other spatially across different scenes in those videos >

Taro: And third is the human-guided base placement refinement where they use the human approach prior to guide docking, inferring direction from how a person positioned themselves before manipulation >

Rosa: So it means we can use that inferred approach direction to find a graspable configuration even when our initial setup is not ideal >

Dev: Fourth is the closed-loop policy revision mechanism, where execution feedback informs the next code generation step, ensuring continuous refinement of navigation and manipulation primitives against live geometry >

Taro: And fifth is the task-independent memory reuse, meaning that knowledge extracted from passive videos isn't specific to one request but serves as a persistent guidance layer for any open-vocabulary instruction >

Rosa: So overall these improvements show that we can extract really rich, reusable spatial and interactive knowledge from everyday videos to guide complex, end-to-end mobile manipulation policies >

Conclusion: Dev: So to wrap up the discussion on MoCaP, it seems the main implication is that passive human video can provide a powerful source of experience for robots without needing specific demonstrations for every task.

Rosa: That's right, and by extracting spatial memory and approach cues, MoCaP provides guidance for both object search and the approach to manipulation >

Taro: It moves beyond just locating things to understanding how interaction is prepared, which is a richer form of knowledge that helps robots operate in unstructured environments >

Dev: The system achieves eighty percent success on two mobile manipulation tasks in an indoor environment, showing it can actually perform these complex actions where existing methods struggle >

Rosa: And the navigation success jumped from seventy percent to one hundred percent, which shows the memory really helps with searching beyond just what’s immediately visible >

Taro: It suggests that this kind of experience can fundamentally change how we think about learning for mobile manipulation because the robot isn't just reacting; it's guided by learned human context >

Dev: We need to keep in mind the limitations, though, like the evaluation being limited to one environment and one video recording, and also that local base search depends heavily on visible geometry and a grasp model >

Rosa: And manipulation is currently limited to grasping and lifting; it doesn't do sustained contact or precise placement yet >

Taro: The paper flags that the system design itself constrains performance because a fixed startup summary can omit relevant objects or become stale, and local base search only depends on visible geometry >

Dev: So the final thought is that while MoCaP shows promise for guiding object search and approach to manipulation from passive human videos, future work needs to evaluate these priors across multiple homes and recordings >

Rosa: That sounds like a good direction; exploring how this works in more diverse settings will really test how generalizable this spatial memory actually is >

Yiyang Cai, Tatsuya Kamijo, Ninghao Lu, Zijin Hu, Sherry Yang

New York University · Fudan University

cs.RO

Submitted: 2026-10-14

Updated: 2026-10-14

Comments: 8 pages, 5 figures, 3 tables. Project page: https://mocap-7.github.io/MoCAP-page/

Code: https://github.com/NVlabs/GraspGenX

Project page: https://mocap-7.github.io/MoCAP-page

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: The gist The MoCaP framework turns passive human video into guidance for an end-to-end mobile manipulation policy by extracting spatial memory and cues of how people prepare to interact with objects,

Key concepts

Spatial Memory: This is the system's way of storing knowledge about a physical place derived from video recordings. It includes descriptions of objects, landmarks, and transition cues like where to turn or what landmark should appear next. This memory helps the robot understand the layout and context of an environment.
Code-as-Policy: This is an agentic framework where a vision-language code agent generates executable Python programs based on current observations and guidance. These programs invoke robot primitives, allowing the system to dynamically coordinate planning, navigation, and tool use in a closed loop.
Human Approach Prior: This is guidance derived from observing how humans position themselves before interacting with an object in videos. This prior informs the robot's approach direction for manipulation tasks. It helps the robot find a reachable base pose or correctly dock by suggesting the right way to get close to the target.

Terminology

Agentic Framework — 1× in this paper · explained in 7 episodes

Summary

The gist The MoCaP framework turns passive human video into guidance for an end-to-end mobile manipulation policy by extracting spatial memory and cues of how people prepare to interact with objects, which informs navigation and the search for a manipulationready configuration.

How it works

MoCaP is an agentic framework that transforms passive human video recordings into guidance for a mobile manipulation policy by extracting spatial memory and approach direction, which inform navigation and the search for a manipulationready configuration The system extracts spatial memory about a place together with records of where people stood when they reached for objects During task execution, the spatial memory helps locate the requested object, and the human approach prior guides the robot to approach the object in the right way A code agent uses this guidance together with current observations to coordinate navigation and manipulation primitives

Offline Spatial and Interactive Memory Construction

The offline builder organizes the recording into text m and timestamped reference frames E, where M = (m,E) = B(V), E = (1) The builder uses GPT-5.6 to generate scene descriptions m that cite relevant video timestamps, then extracts the corresponding reference frames E Together, the text and frames form the spatial memory M The builder links small-object observations to their surrounding regions and records transition cues: where to turn, which landmark should become visible next, and how to recognize a revisited place

Code-as-Policy and Robot-Grounded Primitives

The system implements closed-loop control through a Code-asPolicy module, in which a vision-language code agent generates Python programs that invoke robot-grounded primitives Given the instruction q, startup guidance a0, current visual observation and state summaries ˜ot, and conversation history ht, the agent generates a program ct, which the runtime executes: ct = π(q,a0,o˜t,ht), (rt,ot+1) = ExecuteObserve(ct) Execution feedback rt and a fresh observation inform the next code-generation step The primitives ground video-derived guidance in the robot’s own viewpoint and motion capabilities

Approaching Object under Guidance from Human Prior

The human approach prior guides docking, while the primitives resolve motions using live geometry and robot constraints The resulting bearing provides a human-derived approach prior rather than a fixed robot goal The grasping arm is selected according to the target’s side relative to the robot, with the human hand used to resolve ambiguous cases

Contributions and Evaluation

The contributions include a unified Code-as-Policy system for mobile manipulation that flexibly coordinates navigation and manipulation through executable code Reusable spatial memory from passive human video, which draws on the abundance of everyday human activity recordings to extract object relations, landmarks, and place connectivity for subsequent robot search and navigation Human-guided base placement for manipulation, where the system infers the direction to approach object from how a person positioned themselves before manipulation in passive human videos MoCaP achieves 80% success on each of two mobile manipulation tasks in an indoor environment whereas existing approaches cannot succeed at the task Spatial memory increases navigation success from 70% to 100%, while controlled comparisons suggest that human approach priors improve manipulation readiness from unfavorable initial base configurations Base placement guided by the human prior found a reachable pose in 5 of 5 trials whereas a naive proximity-based docking failed in all five

Limitations and Conclusion

MoCaP has several limitations, including the fact that our evaluation is limited to one environment, one video recording, and three target objects with five to ten trials per condition The system design itself constrains performance because a fixed startup summary can omit relevant objects or become stale Local base search depends on visible geometry and the grasp model and does not optimize future visibility Manipulation is limited to grasping and lifting Future work could evaluate the priors across multiple homes and recordings, and use human video to guide richer skills involving sustained contact, precise placement, or coordinated use of both arms The paper concludes that passive human video can help robots complete mobile manipulation tasks by guiding both object search and the approach to manipulation

--- Page 1 ---

MoCaP: Mobile Code-as-Policies for Zero-Shot Mobile Manipulation from Passive Human Videos Yiyang Cai1,2,∗ Tatsuya Kamijo1,∗, Ninghao Lu3, Zijin Hu1, and Sherry Yang1 Abstract— Passive human videos of everyday activity offer robots access to a massive amount of human experience without requiring a demonstration for each task. However, using this experience requires recovering knowledge from unstructured video recordings and adapting it to a different body (i.e., the robot). To overcome these challenges, we present mobile code-as-policy (MoCaP), an agentic framework that turns passive human video recordings into guidance for an end-to-end mobile manipulation policy. MoCaP extracts spatial memory and cues of how people prepare to interact with objects, which inform navigation and the search for a manipulationready configuration. Guided by passive human experience and current observations, MoCaP uses a closed-loop code-as-policy to coordinate planning and tool use primitives to solve motions under the robot’s geometry and kinematic constraints. On a real mobile manipulator, MoCaP achieves 80% success on each of two mobile manipulation tasks in an indoor environment whereas existing approaches cannot succeed at the task. Videoderived spatial memory improves object-navigation success from 70% to 100%, and human approach prior raises the rate of finding a reachable base pose from 0% to 100% at an unfavorable start location. Videos and code are available on the project website I. INTRODUCTION Household robots should be able to assist different users in a wide range of homes, responding to everyday requests without requiring users to specify every action [1]. Openvocabulary mobile manipulation moves toward this goal by enabling robots to act on natural-language instructions involving diverse objects and environments [1]. To fulfill a request such as “bring the cup back,” a robot must determine which objects and places are relevant, locate them, and decide how to act. Supporting this ability across homes and users motivates learning from experience beyond the robot’s own interactions Passive recordings of human activity offer a potentially abundant source of such experience. Wearable cameras can capture how people interact with objects and their surroundings during everyday life, without requiring robot teleoperation or prescribed demonstrations [2]. We investigate how robots can draw on this accumulated human experience for open-vocabulary mobile manipulation We consider the setting where a collection of human activity videos is passively recorded through people’s daily lives capturing diverse activities that may be requested of a robot. Rather than reproducing a particular demonstration, the robot must use ∗Equal contribution. 1New York University, New York, NY, USA. 2Fudan University, Shanghai, China. Work done while Yiyang Cai was visiting New York University However, passive recordings of egocentric human data do not directly provide the information a robot needs to act Specifically, using passive human experience for robots presents three challenges First, the recorded experience lacks explicit spatial structure Second, human and robot sensing and morphology differ Third, navigation and manipulation are not explicitly separated in human activity MoCaP Efficient Execution Navigation Human Prior Coding Agent Passive Human Videos Accumulate through Spatial memory Approach direction Blue rectangular bin against the orange wall IK IK Fig. 1.

Improvements for AI systems

No explicit separation of navigation and manipulation stages in control: The system employs a unified Code-as-Policy system for mobile manipulation, which flexibly coordinates navigation and manipulation through executable code, without a predefined division into navigation and manipulation stages. This allows the agent to revise actions from both planning domains simultaneously based on feedback.
Enhanced object localization via video memory: The pipeline develops fine-grained spatial memory that draws on the abundance of everyday human activity recordings to extract object relations, landmarks, and place connectivity for subsequent robot search and navigation, which increases navigation success from 70% to 100%. This enables robust searching beyond the current view by connecting objects to their spatial context.
Human-guided base placement refinement: The system uses human approach prior to guide docking, as the recorded interaction occurs on the C side, which helps find a graspable configuration by inferring direction from how a person positioned themselves before manipulation in passive human videos. This guidance can turn arrivals into graspable configurations, using only video that accumulates from everyday activity.
Closed-loop policy revision: The agent utilizes a closed-loop control mechanism where the code generation policy is revised based on execution feedback: Failed action calls interrupt the remaining program, allowing the agent to revise its plan, ensuring continuous refinement of navigation and manipulation primitives against live geometry.
Task-independent memory reuse: The system leverages memory that is task-independent and can be reused to different tasks, meaning the knowledge extracted from passive videos is not specific to a single request but serves as a persistent guidance layer for any open-vocabulary instruction.