Gen4P: Video Generators are 4D Perceivers

arXiv:2610.22371 · cs.CV · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Gen4P: Video Generators are 4D Perceivers".

Jane: The gist: Gen4P introduces a unified framework that repurposes a pretrained video generator into a single-step feed-forward 4D perceiver for recovering full 4D geometry and motion from monocular video.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper, "Gen4P: Video Generators are 4D Perceivers," by Xia, Jiang, and Zheng <ref:2610.22371#pg1,Gen4P: Video Generators are 4D Perceivers>. It's trying to take something that already exists—a video generator—and turn it into a single-step system that can predict everything about a video in four dimensions.

Jane: So what does that mean for us? Basically, it’s aiming to get dense three dee tracks, point maps, depth maps, and camera poses all at once from just one pass through the model without needing any iterative denoising steps <ref:2610.22371#pg1,dense 3D tracks, point maps, depth maps, and camera poses>.

Lu: The core idea is repurposing a multi-step video generator into a feed-forward 4D perceiver without iterative denoising <ref:2610.22371#pg1>. It's about learning a direct regression network instead of doing the iterative denoising that some prior work used for geometric reconstruction.

Meng: So, the paper claims this means only one encoder forward pass is needed per input clip to get scene tokens, and those tokens are shared across different queries for the 4D predictions <ref:2610.22371#pg3>. That sounds efficient in terms of computation.

Lalam: I see how it works: they use an on-demand query formulation to get the full 4D attributes by querying those scene tokens with a lightweight cross-attention decoder <ref:2610.22371#pg1>.

Tom: That part about the on-demand query formulation is interesting because it decouples sampling from pre-extracted feature grids, which gives them spatial flexibility in the output resolution <ref:2610.22371#pg3>.

Jane: And they added continuous-time queries to handle videos of variable lengths effectively, which is a big win for real-world video input.

The paper's summary: Tom: So the paper summarizes Gen4P as a unified framework for recovering 4D geometry and motion from monocular video <ref:2610.22371#pg1>. It shows they can use a pretrained multi-step video generator to predict these four things together.

Jane: The summary highlights that this is achieved by processing clean video latents in a single encoder pass, which then feeds into a cross-attention decoder for the predictions <ref:2610.22371#pg3>.

Lu: They also mention that they introduced continuous-time queries to provide temporal flexibility, parameterizing source time as alpha(tsrc) equals tsrc over T minus one which handles variable-length sequences <ref:2610.22371#pg3>.

Meng: I'm looking at the components here—they use a Video Generative Encoder that has a frozen VAE encoder and DiT blocks, plus an optional frozen Depth-LoRA, and a trainable 4D-LoRA <ref:2610.22371#pg3>. That’s a lot of parts to manage.

Lalam: The paper also points out the plug-in solution they use by incorporating a pretrained video-depth LoRA as a frozen geometry prior, which reuses knowledge from dense depth estimation <ref:2610.22371#pg1>.

Tom: So, to summarize, the main point is that they repurposed the generator into a feed-forward 4D perceiver without iterative denoising <ref:2610.22371#pg1>. It's not just about one part; it’s how they put all these pieces together into this unified structure.

Jane: So, to put it plainly, the paper is saying that a video generator isn't just for pretty pictures anymore; it can be used as a direct tool to reconstruct the full 4D geometry and motion of a scene from video input <ref:2610.22371#pg1>.

The paper's improvements: Tom: Now let's talk about what they improved on top of that—their proposed enhancements to this Gen4P framework. They focused on three main things, specifically how they made it more flexible and accurate.

Jane: First, they tackled the geometry knowledge problem by adding a plug-in solution: a frozen depth LoRA acting as a geometry prior <ref:2610.22371#pg1>. This means they reused knowledge from dense depth estimation without adding extra trainable parameters <ref:2610.22371#pg3>.

Lu: That’s important because the paper admits that geometry priors alone don't explicitly model how points should evolve over time in dynamic scenes, so this generative prior complements those existing geometric methods <ref:2610.22371#pg2>.

Meng: From an engineering standpoint, reusing a frozen LoRA means they skip a whole dense three dee pretraining stage, and the parameter count stays the same as just using the 4D-LoRA alone <ref:2610.22371#pg3>.

Tom: And then there's that continuous-time query mechanism. They parameterized source time as alpha(tsrc) equals tsrc over T minus one which gives temporal flexibility independent of the absolute sequence length T <ref:2610.22371#pg3>.

Jane: That handles variable-length videos really well, and it’s not just about having a formula; it’s about making the query mechanism work for any video duration <ref:2610.22371#pg3>.

Lalam: And they also have specific training objectives that look pretty detailed. They use a composite loss L equals lambda three deeLthree dee plus lambda confLconf plus lambda 2DL2D plus lambda visLvis plus lambda dispLdisp plus lambda normalLnormal <ref:2610.22371#pg1>.

Tom: That's a lot of losses to balance, covering position, confidence, displacement, and surface normals. It’s a comprehensive set of objectives for training <ref:2610.22371#pg4>.

Conclusion: Jane: So if we wrap up the Gen4P paper, the main implication is that we don't need separate, task-specific approaches for 4D perception anymore <ref:2610.22371#pg1>. We can leverage rich priors from large video generators to get full reconstruction in one go <ref:2610.22371#pg2>.

Tom: It really pushes the idea that a foundational generative model can serve as a starting point for unified 4D reconstruction, which is what they demonstrate here <ref:2610.22371#pg1>.

Lu: The continuous-time query and the plug-in geometry prior are key additions because they address flexibility and geometric knowledge that simple priors alone miss <ref:2610.22371#pg3>.

Meng: For deployment, the fact that they can handle sequences up to one hundred five frames without significant performance degradation is a good sign for scalability on real traffic <ref:2610.22371#pg4>.

Lalam: I'm pretty sure their results on Sintel metrics, specifically achieving the lowest errors on all three Sintel metrics among compared methods, show this approach is performing very well in practice <ref:2610.22371#pg4>.

Tom: So that’s the gist of Gen4P: it’s a unified framework built around repurposing a video generator into a feed-forward 4D perceiver <ref:2610.22371#pg1>. It shows that this direction is definitely promising for how we approach 4D reconstruction <ref:2610.22371#pg1>.

Yitong Xia, Zeren Jiang, Chuanxia Zheng

Nanyang Technological University · University of Oxford

cs.CV

Submitted: 2026-10-15

Updated: 2026-10-15

Comments: Project page: https://gen4p.github.io/

Code: https://github.com/jiangyurong609/d4rt-pytorch

Project page: https://gen4p.github.io

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 82/100

The gist: The gist: Gen4P introduces a unified framework that repurposes a pretrained video generator into a single-step feed-forward 4D perceiver for recovering full 4D geometry and motion from monocular

Key concepts

Repurposing the Generator: Gen4P takes a standard video generator, adapted with a LoRA, and reconfigures it to act as a direct 4D perceiver. Instead of needing multiple steps of denoising to reconstruct geometry, it learns to predict all necessary 3D attributes directly from the input video in one go. This simplifies the process significantly by removing iterative noise injection.
Continuous-Time Query: This mechanism allows the model to handle videos of different lengths effectively. It parameterizes time by mapping absolute timestamps onto a normalized position within the clip. This makes the model's temporal queries independent of the total sequence length, giving it flexibility when processing sequences longer or shorter than its training data.
Plug-in Geometry Prior: The model incorporates a pre-trained video-depth LoRA as a fixed geometry prior. This acts like an external knowledge source that is frozen during training. By combining this prior with the main 4D prediction network, Gen4P gains strong geometric knowledge without needing extra trainable parameters for depth estimation.
Flexible Query Decoding: The output sampling process is decoupled from the pre-extracted feature grids. This allows the model to query these features on demand, giving it spatial flexibility in determining the resolution of the predicted 4D attributes. This means it can adapt its output resolution based on what is needed for a specific prediction.

Terminology

Generator — 2× in this paper · explained in 2 episodes
Decoder — 1× in this paper · explained in 2 episodes
Unified Framework — 1× in this paper · explained in 6 episodes

Summary

The gist: Gen4P introduces a unified framework that repurposes a pretrained video generator into a single-step feed-forward 4D perceiver for recovering full 4D geometry and motion from monocular video.

How it works

Gen4P tames a multi-step video generator into a single-step feed-forward 4D perceiver without iterative denoising or noise injection The objective is formulated as learning a direct regression network, instead of the iterative denoising for geometric reconstruction in prior works. This formulation requires only one encoder forward pass per input clip, and the resulting scene tokens are shared across queries, reducing the encoding cost over repeated 4D predictions.
The full 4D attributes are obtained by querying the scene tokens using a lightweight cross-attention decoder. This decouples the output sampling from pre-extracted feature grids, allowing spatial flexibility in the output resolution. Additionally, a continuous-time query mechanism is introduced to handle variable-length video sequences effectively.

Key Components and Enhancements

(i) Repurposing the Generator:

Gen4P utilizes a video-generator-based encoder with a LoRA-adapted video DiT backbone, adapted using a rank-512 LoRA referred to as 4D-LoRA. The input video is processed through the Wan VAE encoder EVAE and DiT blocks FDiT, obtaining scene tokens F as described in Equation (5).

(ii) Continuous Time Query:

The continuous time query mechanism enhances flexibility for variable-length sequences by defining a parameterization for source time tsrc as α(tsrc) = tsrc / T − 1, where esrc(tsrc) is defined as (1 − α(tsrc)) esrc,0 + α(tsrc) esrc,1. This parameterization represents each timestamp by its normalized position within the clip, making the time query independent of the absolute sequence length T and thereby enhancing the model’s temporal flexibility.

(iii) Plug-in Geometry Prior:

To strengthen geometric knowledge, Gen4P incorporates a plug-in solution by incorporating a pretrained video-depth LoRA as a frozen geometry prior. This is achieved through a double-LoRA architecture where the effective weight is W = WDiT + ∆W512 Depth + ∆W512 4D. This approach reuses knowledge learned from dense depth estimation without adding extra trainable parameters, and the double-LoRA variant requires no additional dense prediction head and has the same number of trainable parameters as the variant using 4D-LoRA alone.

Training Objectives and Losses

The training objective is described by a composite loss L = λ3DL3D + λconfLconf + λ2DL2D + λvisLvis + λdispLdisp + λnormalLnormal. The specific losses are defined as follows:

(C1) 3D Position and Confidence:

The per-query 3D error is defined as the signed logarithmic transform ϕ(x) = sign(x) log(1 + x) element-wise to reduce the influence of distant points, with the per-query 3D error being L3D = ⟨cil3D,i⟩m3D.

(C2) 2D Position and Visibility:

The 2D position loss is applied using an element-wise Huber loss with unit transition point: L2D = X d∈[x,y] Huber1 pˆi,d − p GT i,d + m2L2D. Visibility is supervised using binary cross-entropy: Lvis = BCEvˆi, vGT i.

(C3) 3D Displacement and Surface Normal:

The displacement loss uses the L1 norm for motion from tsrc to ttgt in the camera frame specified by tcam: Ldisp = D ∆Pˆi − ∆P GT i / E mdisp. The surface normal loss supervises angular discrepancy using cosine distance: Lnormal = 1 − nˆi∥nˆi∥2⊤ n GT i∥n GT i∥2 + mnormal.

Evaluation and Results

Gen4P achieves state-of-the-art or competitive performance on tasks of 3D point tracking, video depth and camera pose estimation. Qualitative results on in-the-wild videos illustrate its capability of tracking complex motion while maintaining the delicate structure. The method outperforms prior methods by a clear margin on DR and PS, where PS is a challenging, fully dynamic human–object benchmark on which no method has been trained with comparable data. Furthermore, the geometry prior further improves every reported tracking metric over the basic variant. Gen4P naturally supports high-resolution depth estimation by querying on high-resolution query frames.

Gen4P demonstrates flexibility in handling variable lengths, from sequences within the 49-frame training length to substantially longer sequences of up to 105 frames, without significant degradation in performance. The method also exhibits less background motion leakage and a clearer motion boundary when analyzing displacement maps compared to baseline methods. Gen4P achieves the lowest errors on all three Sintel metrics among the compared methods (Tab. 3). Finally, the results demonstrate that video generators are a promising foundation for unified 4D reconstruction.

The paper is supported by NTU SUG-NAP and National Research Foundation, Singapore, under its NRF Fellowship Award NRF-NRFF17-2025-0009. The authors used LLM-based AI tools to assist with grammar correction, language polishing, and adjustments to figure and table layouts in the manuscript. The authors assume full responsibility for the final manuscript, including its scientific claims and reported results. All training datasets are publicly available, and the Wan2.1 backbone (Wan et al., 2025) and the depth LoRA (Zhang et al., 2026b) have been publicly released. The paper is reproducible by describing the model architecture, continuous-time query, plug-in geometry prior, and training objective in Section 3. The authors report results on approximately 200K video clips from 11 public datasets for training. All variants are trained for 45K steps at an effective batch size of 16, on a mixture of DR, PO, Kubric CoTracker and Syn4D with an encoder frame resolution of 256 × 512 and a 6-layer decoder. The learning rate is linearly warmed up to a peak of 10−4 over the first 2.5K steps and then follows a cosine decay schedule to 10−6. All training datasets are publicly available, and the Wan2.1 backbone (Wan et al., 2025) and the depth LoRA (Zhang et al., 2026b) have been publicly released. The paper is reproducible by describing the model architecture, continuous-time query, plug-in geometry prior, and training objective in Section 3. All training datasets are publicly available, and the Wan2.1 backbone (Wan et al., 2025) and the depth LoRA (Zhang et al., 2026b) have been publicly released. The paper is reproducible by describing the model architecture, continuous-time query, plug-in geometry prior, and training objective in Section 3. All training datasets are publicly available, and the Wan2.1 backbone (Wan et al., 2025) and the depth LoRA (Zhang et al., 2026b) have been publicly released. The paper is reproducible by describing the model architecture, continuous-time query, plug-in geometry prior, and training objective in Section 3. All training datasets are publicly available, and the Wan2.1 backbone (Wan et al., 2025) and the depth LoRA (Zhang et al., 2026b) have been publicly released. The paper is reproducible by describing the model architecture, continuous-time query, plug-in geometry prior, and training objective in Section 3. All training datasets are publicly available, and the Wan2.1 backbone (Wan et al., 2025) and the depth LoRA (Zhang et al., 2026b) have been publicly released. The paper is reproducible by describing the model architecture, continuous-time query, plug-in geometry prior, and training objective in Section 3. All training datasets are publicly available, and the Wan2.1 backbone (Wan et al., 2025) and the depth LoRA (Zhang et al., 2026b) have been publicly released.

Improvements for AI systems

Bold header: Unified 4D Perception Framework Development

Gen4P repurposes a multi-step Video Generator into a single-step 4D Perceiver, enabling unified feed-forward prediction of dense 3D tracks, point maps, depth maps, and camera poses from a video input. This allows the system to perform simultaneous reconstruction of all four modalities in one forward pass without iterative denoising.

Bold header: Enhanced Temporal Flexibility

The introduction of the continuous-time query mechanism allows the model to handle real-world videos of variable lengths by parameterizing timestamps as normalized position within the clip, which provides temporal flexibility independent of absolute sequence length T.

Bold header: Geometry Prior Injection

Incorporating a frozen depth LoRA enables a plug-in geometry prior that improves overall performance without adding extra trainable parameters, effectively allowing the system to reuse knowledge learned from dense depth estimation without a separate dense 3D pretraining stage.

Bold header: Flexible Query Decoding

The use of an on-demand query formulation decouples the output sampling from pre-extracted feature grids, which enables spatial flexibility in the output resolution, allowing for variable spatial sampling in the predicted 4D attributes.

Bold header: Robust Motion Boundary Detection

By comparing background motion with foreground boundary analysis, Gen4P exhibits less background motion leakage and a clearer motion boundary, leading to more visually plausible tracking results, such as better preservation of delicate structures.