Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation

arXiv:2610.22363 · cs.CV, cs.RO · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Perceive, Predict, and Plan".

Jane: The gist The P3-WAM model introduces a unified world action model for egocentric dexterous manipulation that learns 3D perception, action-conditioned prediction, and goaldirected planning through three complementary modes: PERCEIVE,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, we're looking at this paper today, "Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation." It sounds like they’re trying to connect three separate areas of research into one system.

Jane: It seems like the core idea is about how an AI needs to handle three specific tasks when it's interacting with something in the real world, which are perceiving what happened, predicting what will happen next, and then planning what action to take.

Lu: Exactly. They’re tackling those three fundamental questions: What happened in an observed interaction? What will happen under a specified action? And what should be done to accomplish a desired goal?

Lu: It’s about linking perception, prediction, and planning together through the knowledge of actions and their consequences.

Meng: From my side, I'm curious how they actually build this system. If you have separate models for each part, it can get really messy when you try to make them talk to each other correctly.

Tom: Well, the paper says they built P3-WAM on a pretrained video diffusion transformer using a mixture-of-transformers architecture that shares base weights but has adaptations for different types of data.

Tom: They’re representing each action by encoding the articulated poses of both hands and the camera pose as compact tokens alongside the video tokens, which is how they connect everything.

Jane: That sounds like they are trying to make sure that when you look at a video, you aren't just seeing pixels, but also knowing exactly where the hands and cameras are in three dimensions.

Jane: They call this specific mode PERCEIVE—reconstructing articulated hand motion and camera trajectories from video.

Lu: And they claim that for the PERCEIVE mode, they achieved the lowest presence-aware reconstruction error on all four datasets, with an average camera-space MPJPE of sixty-three point zero five millimeters <ref:2610.22363#pg7>. That's a pretty strong metric for getting that three dee motion right from just a video feed <ref:2610.22363#pg1>.

Meng: But reconstruction is only half the battle, right? If we can't predict what happens next when we actually move, it’s just a static picture of what happened.

Title and authors: Tom: Right, and they have the PREDICT mode which teaches the model to anticipate visual consequences of prescribed actions.

Tom: The training method here is interesting because they randomly vary the visibility of visual observations and actions across clips and time to train all three modes simultaneously.

Jane: So, when you’re watching a video, sometimes you're shown the video but not the action, which helps with perception, and sometimes you’re shown the action but not what happens next in the future.

Jane: They use this mixed training approach to get these different skills working at once.

Lu: The PREDICT mode improved prediction performance quite a bit; they saw PSNR increase from seventeen point three five six dB to eighteen point one three eight dB and FVD decrease from one thousand one hundred five point nine five to one thousand nine point eight four <ref:2610.22363#pg6>. They also managed to make the action branch predict the full hand and camera trajectory in just one forward pass instead of needing iterative denoising steps <ref:2610.22363#pg4>.

Meng: One forward pass is nice for speed, but what about when you have a real robot running this? Does that single pass hold up when the environment changes unexpectedly?

Tom: That brings us to the PLAN mode, which is where they jointly generate action trajectories and their anticipated visual consequences guided by text instructions or an optional visual goal.

Tom: This is the planning side of things, where it’s all put together.

Jane: And this joint generation capability seems really powerful for goal-directed control. They tested this in the PLAN mode, and they saw gains in VLM scores across all five dimensions when unified training was used <ref:2610.22363#pg6>.

Lu: The largest gains they found were in temporal/identity consistency, which jumped from two point one five to three point five seven, and goal transition, which went from one point nine eight to three point two nine <ref:2610.22363#pg6>. That shows the instruction following is getting much more reliable when they use these visual subgoals during planning <ref:2610.22363#pg1>.

Meng: So, if I had to deploy this, what would be my biggest worry? Is it the complexity of training or how stable it is when we try to run it on actual traffic data?

Tom: The unified training framework seems to reinforce all three modes better than specialized baselines across all three tasks <ref:2610.22363#pg2>. It shows that mixing the inputs—video, actions, and instructions—is what actually makes the model perform well across perception, prediction, and planning.

Title and authors: Jane: But there are still some sticking points they mentioned in their conclusion. They noted that there is an inherent ambiguity in estimating global three dee hand position from monocular video <ref:2610.22363#pg9>. That means even with all the learning, getting a perfectly accurate global view of the hands from just one camera can be tricky.

Lu: And they also pointed out that for real-world robot experiments, they are relying on open-loop execution <ref:2610.22363#pg9>. That means we aren't testing it in a fully controlled, closed loop environment yet; it needs to be executed in the real world to see if the generated behavior captures accurate object interactions.

Meng: I agree with Lu on that execution point. If the model can generate perfect movements in simulation, but when you put it on a physical robot, something breaks because of latency or unexpected friction, that’s a deployment hurdle.

Tom: So, to wrap up: P3-WAM successfully integrates perception and prediction and planning into one unified model for egocentric dexterous manipulation <ref:2610.22363#pg5>. It shows the power of training across those three complementary modes together.

Jane: It really shows how linking knowledge of actions and their consequences helps an AI move beyond just mapping pixels to actions, toward true goal-directed control in complex physical tasks.

Lu: And it suggests that we might not need entirely separate, isolated models for reconstruction and prediction if we can train them together with varied inputs <ref:2610.22363#pg2>.

Meng: For deployment right now, it means we need to keep an eye on how robust these unified models are when the inputs aren't perfectly clean or when the required precision for global hand position is too high for a standard camera setup.

Tom: Exactly. So, the P3-WAM paper gives us a single framework that handles perception, prediction, and planning together, even with those practical limitations regarding three dee estimation ambiguity and reliance on open-loop execution <ref:2610.22363#pg9>.

Jane: It’s an exciting step toward building embodied AI systems that can truly understand goals like a human does by observing and interacting with the world around them.

The paper's summary: Tom: So, we’re looking at P3-WAM today, and the main thing is that they put three separate skills—perception, prediction, and planning—into one single model for manipulating objects in the real world.

Jane: Right! The paper boils it down to three modes: PERCEIVE, which is about reconstructing three dee motion from a video; PREDICT, which anticipates what will happen when you take an action; and PLAN, which plans the action trajectory based on a goal and some instructions.

Lu: What’s really interesting for me is how they force the model to learn all three at once through this unified training framework where they randomly hide things—sometimes video without actions, sometimes actions without future video, and sometimes both hidden—to train each mode specifically.

Meng: I mean, from a deployment standpoint, that mixed training setup sounds smart because it’s trying to give the model a really broad set of experiences across all those skills simultaneously. But what does that actually mean for the performance when we run this stuff on actual robots?

Tom: It means they aren't just training three separate specialists; they’re creating one system where learning to perceive helps with predicting, and prediction helps with planning, all from the same starting point. They show that this unified training improves all three capabilities over just doing one mode separately.

Jane: The results on the PLAN mode are pretty compelling for goal-directed control. They showed big jumps in temporal consistency and goal transition scores when they used those visual subgoals during planning, which means it’s much better at following complex instructions to reach a target.

Lu: I saw the numbers on that: the goal transition score went up from one point nine eight to three point two nine when they added those visual goals during unified training <ref:2610.22363#pg6>. That shows that linking perception and planning through those subgoals really helps structure the output better.

Meng: But I have to ask, what about the limitations? If this model is so complex, how much computational power do we actually need to run it when we try to scale it up for a real robot deployment?

Tom: The authors are honest about that. They flag two main things: first, there's still an inherent ambiguity in estimating the global three dee hand position just from one camera video. Second, they say their real-world experiments rely on open-loop execution, meaning we aren't testing it in a fully controlled environment yet.

Jane: That means even though the model is getting better at reconstructing hand motion and planning actions, we still have to be careful about how accurately it can map that three dee pose globally without more information.

Lu: It also highlights that while the model learns from video diffusion priors, transferring that human-hand motion to a physical robot still requires those real-world deployment tests where they are open-loop.

Meng: So, for someone who just listens to this show and doesn't read the technical details, what does this mean? It means we have a single model that handles seeing, predicting, and planning for dexterous manipulation all at once.

Tom: Exactly. It’s like giving an AI a whole toolkit instead of three separate tools; it learns how to use the perception to guide the prediction to inform the plan.

Jane: And that's what makes this model different from previous work, because it proves that linking those three stages through unified training actually gives you better performance across all those tasks than having them operate in isolation.

Lu: It suggests that for complex tasks like dexterous manipulation, the real power isn't necessarily inventing a single perfect perception module or a single perfect planner; it’s how effectively you link those capabilities together.

Meng: So, the big picture is that we have this unified structure, but we still have to watch those challenges—the ambiguity in three dee position and the need for real-world testing before we can fully trust it on a robot.

Tom: Right. We see how linking knowledge of actions and their consequences helps an AI move beyond just mapping pixels to actions, toward true goal-directed control in complex physical tasks.

Jane: It’s a big step toward building embodied AI systems that can truly understand goals like a human does by observing and interacting with the world around them.

The paper's improvements: Tom: So, moving past just the basic framework, P3-WAM shows real gains in performance when you use that unified training setup compared to models that only focus on one of those three modes alone.

Jane: Right! It’s not just about having these three parts; it’s about how they work together—the paper says this unified training actually improves all three capabilities over single-mode training.

Lu: I mean, the gains in planning quality under a vision-language model evaluation are significant, specifically when they look at instruction adherence and physical interaction dimensions.

Meng: Can you give me some of those numbers? I need to know what that improvement actually looks like on the ground when we talk about deployment readiness.

Tom: They showed improvements in VLM scores across all five dimensions. The biggest ones were in temporal/identity consistency, which went from two point one five to three point five seven <ref:2610.22363#pg6>. And goal transition score improved from one point nine eight to three point two nine <ref:2610.22363#pg6>.

Jane: Those numbers are concrete, showing that the system gets much better at following instructions and keeping track of what’s happening over time when it’s doing the planning part.

Lu: And they also improved prediction performance significantly; PSNR went up from seventeen point three five six decibels to eighteen point one three eight decibels <ref:2610.22363#pg6>, and FVD went down from one thousand one hundred five point nine five to one thousand nine eighty nine point eight four <ref:2610.22363#pg4>.

Meng: That’s good for the prediction side, but what about the reconstruction? I heard that was also a big focus. If it can’t accurately see what’s there first, the planning part doesn't matter much.

Tom: They did improve hand reconstruction too. They achieved lower presence-aware reconstruction error on all four datasets compared to models like HaMeR and DynHaMR <ref:2610.22363#pg7>.

Jane: So, when we look at the overall picture, what does this mean for someone who just listens to the show? It means the system is becoming much more reliable at figuring out where hands are in three dee space and what motion they’ll make next based on a goal.

Lu: It suggests that by learning all these modes together, you create a more robust representation of the world interaction than if you just trained them separately.

Meng: The implication for us deploying this is that we need to be careful about those real-world execution tests they mentioned, because while the generation looks good in simulation, we still have to verify it on actual robots.

Tom: Exactly. It’s a win for integration, but the authors are clear: there’s still that ambiguity in estimating global three dee hand position from monocular video <ref:2610.22363#pg9>.

Jane: So, the main message is this: P3-WAM proves you can integrate perception, prediction, and planning into one model for manipulation, but you still have to deal with those inherent challenges of three dee estimation and relying on open-loop execution for real robot work.

Conclusion: Tom: So we’ve been going through "Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation," and the big picture is that they successfully put three separate skills—perception, prediction, and planning—into one single system for manipulating things in the real world.

Jane: Right! It’s about linking those three stages together so that when an AI sees something, it knows what happened and can then plan how to do something else to reach a goal.

Lu: I think the main thing they achieve is showing that this unified training approach actually gives you better performance across all those tasks than if you just trained them in isolation.

Meng: So, when we look at the overall picture, what does this mean for someone who only listens to the show? It means we have a single model that handles seeing, predicting, and planning for dexterous manipulation all at once.

Tom: Exactly. It’s like giving an AI a whole toolkit instead of three separate tools; it learns how to use the perception to guide the prediction to inform the plan.

Jane: And that’s what makes this model different from previous work, because it proves that linking those capabilities through unified training actually gives you better performance on reconstruction, prediction, and planning tasks.

Lu: It suggests that for complex tasks like dexterous manipulation, the real power isn't just inventing a single perfect perception module or a single perfect planner; it’s how effectively you link those capabilities together.

Meng: I still have to be careful about those real-world execution tests they mentioned, because while the generation looks good in simulation, we still have to verify it on actual robots.

Tom: Right. They were clear that there’s still that ambiguity in estimating global three dee hand position from monocular video and they flagged the need for open-loop execution for robot experiments.

Jane: So, while it’s a big step toward building embodied AI systems, we still have to deal with those challenges of three dee estimation and relying on open-loop execution before it's fully ready for the factory floor.

Lu: It’s a win for integration, but the authors are clear that their framework is built on a video diffusion transformer architecture that shares weights across modalities, which is a smart way to draw on existing visual priors.

Tom: So, to wrap up, "Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation" shows how linking knowledge of actions and their consequences helps an AI move toward true goal-directed control.

Jane: It really shows how the field is moving toward models that can truly understand goals by observing and interacting with the world around them, even if there are still some technical hurdles to clear.

Lu: I think this paper opens up a lot of avenues for future work, especially in how they handle those representation tokens and making sure that compression doesn't lose critical visual information.

Meng: We’ll be watching how they tackle the scaling issues we talked about, because if you can run this on real traffic data without it breaking at scale, then this whole idea becomes much more practical for production.

Tom: Next time we talk about a paper that tackles scaling or representation efficiency, we’ll see where P3-WAM fits into that picture.

Zirui Wu, Chen Guo, Tianjian Jiang, Zeren Jiang, Zeming Li, Lingchen Yang, Longteng Duan, Ying Xue, Handi Yin

HKUST University of Science and Technology Hong Kong Special University of Science and Technology Zurich ETH Zurich Visual Geometry Group University of Oxford University of St. Gallen

cs.CV, cs.RO

Submitted: 2026-10-15

Updated: 2026-10-15

Code: https://github.com/Wan-Video/Wan2.2

Project page: https://p3-wam.pages.dev

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: The gist The P3-WAM model introduces a unified world action model for egocentric dexterous manipulation that learns 3D perception, action-conditioned prediction, and goaldirected planning through

Key concepts

PERCEIVE Mode: This mode focuses on 3D interaction reconstruction. The model recovers articulated hand motion and camera trajectories from observed videos. It aims to answer 'What happened in an observed interaction?' achieving low reconstruction errors by learning from video data while withholding action information.
PREDICT Mode: The PREDICT mode teaches the model to anticipate visual consequences of prescribed actions. It generates future visual states under specific actions. Unified training improves this by showing the model how to predict full hand and camera trajectories in a single forward pass, rather than through iterative denoising.
PLAN Mode: This mode enables joint action–future planning guided by text instructions and optional visual goals. The model generates sequences of actions along with their anticipated visual outcomes. This capability allows the system to answer 'What should be done to accomplish a desired goal?' using both language and visual cues.
Unified Training Framework: This framework trains the model across all three modes simultaneously by randomly varying the visibility of visual observations and actions during training. This forces the model to learn complementary skills: video-only input supports perception, action-only input supports prediction, and both support joint planning.

Terminology

Transformer — 2× in this paper · explained in 2 episodes
Adaptation — 1× in this paper · explained in 1 episode
Transformers — 1× in this paper · explained in 2 episodes

Summary

The gist The P3-WAM model introduces a unified world action model for egocentric dexterous manipulation that learns 3D perception, action-conditioned prediction, and goaldirected planning through three complementary modes: PERCEIVE, PREDICT, and PLAN.

How it works

P3-WAM integrates three complementary modes: PERCEIVE, which reconstructs articulated hand motion and camera trajectories from video (Page 1), PREDICT, which generates visual futures under prescribed actions (Page 1), and PLAN, which jointly generates action trajectories and their anticipated visual consequences conditioned on an initial image, a language instruction, and an optional visual goal (Page 1). The model is built on a pretrained video diffusion transformer using a mixture-of-transformers architecture with shared base weights and modality-specific adaptation (Page 2). This unified training is achieved by randomly varying the visibility of visual observations and actions: providing video while withholding actions supports perception, providing actions while withholding future video supports prediction, and withholding actions and video supports joint action–future planning (Page 2).

Unified Training Framework

The unified training framework exposes the model to complete modes and partially observed interactions by sampling visibility patterns per clip and across temporal positions (Page 2). The model represents each action by the articulated poses of both hands and the camera pose, and encode these states as compact tokens alongside the video tokens (Page 2). The architecture couples video and action streams through joint attention, where the action expert adds low-rank updates to these six projections (Page 3). The training objective is a total loss L = Lvideo + Lparam + Lspatial + Ltemporal + Lpresence + Lcamera (Page 3).

PERCEIVE Mode

The PERCEIVE mode focuses on 3D interaction reconstruction, where the model recovers 3D motion from observed interactions (Page 2). The model achieves the lowest presence-aware reconstruction error on all four datasets, with an average camera-space MPJPE-p of 63.05 mm (Page 7). Qualitative results show that our predictions show more consistent hand placement and articulation, while HaMeR and Dyn-HaMR miss or misplace hands, and left/righthand confusion (Page 7).

PREDICT Mode

The PREDICT mode teaches the model to anticipate visual consequences of prescribed actions (Page 2). Unified training improves prediction performance, with PSNR increasing from 17.356 to 18.138 dB and FVD decreasing from 1105.95 to 1009.84 (Page 6). The model's action branch predicts the full hand and camera trajectory in one forward pass, rather than iterative denoising (Page 4).

PLAN Mode

The PLAN mode jointly generates video and trajectories guided by text instructions and visual goals (Page 1). Unified training improves VLM scores across all five dimensions, with the largest gains are in temporal/identity consistency (2.15 to 3.57) and goal transition (1.98 to 3.29) (Page 6). When a goal image is supplied, the model reconstructs the relevant objects and scene and recover their motion over time (Page 26).

Evaluation

The model's performance is evaluated across three main areas:

PERCEIVE evaluation compares video-based 3D hand reconstruction against HaMeR and Dyn-HaMR, where P3-WAM achieves the lowest presence-aware reconstruction error on all four datasets (Page 7).
PREDICT evaluation compares action-conditioned video prediction against baselines like Wan2.2 and DreamDojo, where P3-WAM achieves the best full-frame results on all four datasets (Page 8).
PLAN evaluation uses a method-blind VLM judge to rate action process, instruction adherence, physical interaction, temporal/identity consistency, and goal transition (Page 9).

The model's capabilities are further tested by comparing its generated videos with baselines and through real-world execution where the generated behavior captures sufficiently accurate object interactions and action ordering to support downstream robot execution (Page 21). The unified training framework is demonstrated to reinforce the three modes, outperforming specialized baselines across all three tasks (Page 2). The model's ability to transfer human-hand motion to a dexterous robot is also illustrated through Robot demonstrations illustrate how generated human-hand motion can be transferred to a dexterous robot (Page 2). The authors conclude that P3-WAM successfully integrates perception, prediction, and planning into a single model (Page 5). The limitations noted are the inherent ambiguity in estimating global 3D hand position from monocular video and the reliance on open-loop execution for robot experiments (Page 9).

--- Page 1 ---

Preprint

PERCEIVE PREDICT Visual subgoal

initial image + instruction

P3 -WAM

→ Video + hand/camera trajectories

--- Page 2 ---

1 INTRODUCTION

Dexterous manipulation requires more than mapping pixels directly to actions. An embodied model must answer three complementary questions: What happened in an observed interaction? What will happen under a specified action? What should be done to accomplish a desired goal? These questions correspond to three fundamental capabilities: perceiving the 3D motion underlying observed interactions, predicting action-conditioned world dynamics, and planning actions together with their anticipated consequences (Page 2). Infants can use what they learn from observing adults’ actions to guide their own behavior (Hauf & Aschersleben, 2008). Through their own interactions, they learn to predict action outcomes and use those expected outcomes to select actions (Verschoor et al., 2013) and organize movements toward a goal (Claxton et al., 2003). Together, these findings suggest that perception, prediction, and planning are linked by knowledge of actions and their consequences (Page 2). Despite these connections, prior work on egocentric manipulation has developed along three closely related but largely separate lines. Egocentric reconstruction methods recover structured 3D interaction information, such as hand motion and camera trajectories, from observed videos (Pavlakos et al., 2024; Yu et al., 2025). Action-conditioned world models predict future observations under prescribed motion trajectories (Hao et al., 2026; Wang et al., 2026b; Sun et al., 2026). World action models jointly generate actions and future observations for goal-directed control (Ye et al., 2026). These capabilities, however, are still largely learned in isolation or coupled only in subsets (Page 2). To close this loop, we introduce P3-WAM, a unified world action model for egocentric dexterous manipulation that integrates three complementary modes: 3D interaction reconstruction (PERCEIVE), action-conditioned world prediction (PREDICT), and joint action–future planning (PLAN) (Page 2). Reconstruction develops the model’s capacity for 3D perception by recovering 3D motion from observed interactions. Prediction teaches the model to anticipate the visual consequences of prescribed actions. Together, these capabilities provide the foundation for world–action planning, in which the model learns to jointly generate goal-directed hand action trajectories and their anticipated visual futures (Page 2). We learn all three modes through unified training. Figure 1 illustrates these three modes (Page 2). We build P3-WAM on a pretrained video diffusion transformer (Wan et al., 2025) to draw on its rich visual and motion priors. We represent each action by the articulated poses of both hands and the camera pose, and encode these states as compact tokens alongside the video tokens (Page 2). To train this architecture across all three modes, we randomly vary the visibility of visual observations and actions: providing video while withholding actions supports perception, providing actions while withholding future video supports prediction, and withholding actions and video supports joint action–future planning (Page 2). Over long sequences, successive visual subgoals specify the target arrangement for each planning window (Page 2).

--- Page 3 ---

2 RELATED WORK

Video World Models and World Action Models. Action-conditioned world models predict single images (Sudhakar et al., 2024) or roll out videos (Akkerman et al., 2025; DreamX Team et al., 2026; 1). Monocular Hand Reconstruction. Recovering 3D hand motion from video requires estimating hand articulation while accounting for a moving camera. Single-image methods provide hand pose and shape estimates: HaMeR (Pavlakos et al., 2024) scales transformer capacity and heterogeneous training data, while WiLoR (Potamias et al., 2025) integrates localization, handedness estimation, and image-aligned mesh refinement. For egocentric sequences, HaWoR (Zhang et al., 2025) and DynHaMR (Yu et al., 2025) combine hand-motion estimation with SLAM to recover global trajectories, using temporal infilling or interacting-hand priors and optimization. Concurrent work MINT (Zhu et al., 2026) jointly estimates camera and hand motion. Video generative models offer another source of information about hand motion. Concurrent works ViDiHand (Wang et al., 2026a) and ACE-Ego-Hand (Liu et al., 2026) use video diffusion features for hand motion recovery.

Improvements for AI systems

Improved Egocentric Dexterous Manipulation via P3-WAM: The unified training framework improves all three capabilities over single-mode training, including gains in planning quality under a vision–language model evaluation. This allows the system to achieve higher VLM scores across action process, instruction adherence, physical interaction, temporal / identity dimensions when performing goal-directed manipulation.
Enhanced 3D Hand Reconstruction: The model achieves the lowest presence-aware reconstruction error on all four datasets, meaning it can more accurately recover hand motion from observed videos compared to baselines like HaMeR and DynHaMR, specifically reducing camera-space MPJPE by up to 23.46%.
Superior Action-Conditioned Prediction: The system outperforms specialized baselines on 3D hand reconstruction and action-conditioned video prediction across four datasets, leading to an increase in PSNR from 17.356 to 18.138 dB and a decrease in FVD from 1105.95 to 1009.84 under unified training conditions.
Goal-Directed Planning with Visual Subgoals: The model can generate manipulation videos with stronger instruction adherence and action progression when using visual subgoals, as evidenced by the gain in goal transition score from 1.98 to 3.29 in unified training settings over PLAN-only baselines.
Transferable Robot Manipulation: The system enables the transfer of human-hand motion to a robot by generating trajectories that are grounded in physical execution, as illustrated by the successful robot deployment experiments where generated human motion is retargeted for real-world execution.