Dense4D: Pixel-Level 4D Human Correspondence

The gist: Dense4D introduces a novel feed-forward model for dense 4D human correspondence across body, clothing, and hair from monocular RGB images without requiring a body template.

How it works

Dense4D operates through a top-down approach where it isolates each person in the scene and jointly processes their crops across time. It predicts Human Point Maps (HPMs) and depth for each image, which assign each person pixel the 3D position of its observed surface point at a common reference time. These HPMs support amodal correspondence, allowing points hidden in the reference image to be localized from observations in other frames. Matching HPM coordinates across crops establishes correspondence without predicting camera poses or large scene displacements, and per-image depth lifts these matches into 3D tracks.

The model utilizes an image encoder and aggregator architecture of Wang et al. (2026), where an additional crop token conditions the aggregator to alternate attention within and across frames. This architecture allows two dense heads to predict HPMs and per-image depth in a single forward pass. The training uses human geometry and correspondence supervision from synthetic motion and multi-view renderings of real scans.

Key Methodologies

1. Human Point Map (HPM) Prediction: The HPM target Xt(p) is the position at the reference time of the surface point observed at p in frame t, expressed in the reference image’s virtual camera coordinates. The scalar depth target Dt(p) is the same point’s optical-axis depth in frame t’s virtual camera at time t.

2. Crop-Aware Prediction: The model constructs image intrinsics Kt from focal length ft, and a crop embedding supplies geometry information to condition the aggregator. This conditioning is achieved by mapping the crop token to one additional token per image before the first attention block.

3. Correspondence Estimation: Instead of predicting point motion in a shared camera frame, Dense4D identifies corresponding pixels by matching HPM coordinates across person-centred crops. Each matched pixel is then lifted with the depth predicted for its image to recover the 3D track position.

4. Tracking and Reconstruction: For a query pixel p in frame t, Dense4D searches only the HPM samples belonging to the person mask in target frame u. A match is estimated as visible when the residual at qˆ is below τ = 0.005. The predicted depth then lifts the matched pixel into the target frame’s original camera coordinates to produce 4D tracks.

Training and Supervision

The model is trained with animated surfaces from BEDLAM2.0, multi-view renderings of static THuman2.1 scans, and MammaSyn body-surface tracks. For supervised pixels i = (t, p) with p ∈ Mt, the point loss Lpt and depth loss Ldp are minimized with equal weight. The confidence-weighted objective follows DUSt3R and VGGT (Wang et al., 2024a; 2025a).

Evaluation Protocols

Dense4D is evaluated through point tracking on real moving people using the TAPVid-3D Panoptic Studio benchmark. It also undergoes an amodal 3D localisation evaluation on 4DDRESS, measuring how closely observations across time align with the full reference-time scan. Performance metrics include Jaccard (AJ), average percentage of points within depth-adaptive thresholds (APD), and occlusion accuracy (OA). Dense4D achieves 38.4 AJ, 50.3 APD, and 89.6 OA on Panoptic Studio Humans. In the amodal evaluation on 4D-DRESS, the mean one-sided point-to-scan distance is less than 1 cm (0.86 cm), compared to baselines like V-DPM (2.25 cm) and 4RC (8.39 cm). Furthermore, Dense4D shows superior human surface consistency on DAVIS, where its transferred points follow the moving person more faithfully than V-DPM and 4RC.

Ablations and Analysis

Ablation studies show that removing the entire crop embedding increases BEDLAM2 pointwise reconstruction error from 2.99 to 4.06 cm and reduces Panoptic Studio Humans tracking AJ from 38.4 to 30.4. Retaining crop zoom while fixing the three centre-ray channels to (0, 0, 1) at inference yields equivalent reconstruction and tracking performance at the reported precision. Temporal analysis on BEDLAM2 shows that extending the interval from one to sixteen seconds increases error from 2.5 to 3.6 cm.

Conclusion

Dense4D recovers dense 4D correspondence across the body, clothing and hair without a body template. It decouples correspondence estimation from camera-pose and scene-motion recovery, supporting direct matching across long temporal intervals. The model produces more precise trajectories and a better fit to measured human geometry than evaluated baselines. Dense4D provides a foundation for augmenting scene understanding systems with more faithful human geometry, dense correspondence and robust tracking.

Limitations

Self-contact can make reference-space correspondence ambiguous when distinct surface points coincide. A broader challenge is evaluating correspondence on real people, as 4D-DRESS measures proximity to scans rather than point identity. Synthetic data provides exact correspondence, but domain differences limit its relevance to real images. The evaluation of scene integration remains open as well.

Future Work

Future work could involve temporal and multi-view consistency losses on large corpora of real video to improve correspondence and reduce reliance on synthetic supervision. Developing real-data benchmarks with independently measured surface-point identities would strengthen evaluation beyond surface proximity and reconstruction-derived tracks. The alignment provided by Dense4D could support learning appearance models based on Gaussian splatting from humans in motion. Its accurate tracks could control human motion in generated video. Dense4D provides a foundation for augmenting scene understanding systems with more faithful human geometry, dense correspondence and robust tracking.

Use of Large Language Models

LLM-based tools assisted with implementing components of the system, running and organizing experiments, and writing scripts for visualizations using 3D graphics tools. They also assisted with editing and polishing the manuscript and occasionally supported literature and dataset searches.

Ethics Statement

This work uses existing research datasets and publicly available video footage, including clips from Pexels (n.d.), in accordance with their respective terms of use. No new recordings of people were collected. Training uses BEDLAM2, THuman2.1 and MammaSyn; evaluation includes existing benchmarks of real videos and 3D scans. Dense4D estimates human geometry and correspondence without predicting biometric identity. Applications should respect the consent and privacy of the people depicted. Our experiments do not assess demographic differences in performance or suitability for consequential decisions about individuals. The underlying datasets and pretrained models remain subject to their respective terms of use.

Reproducibility Statement

The representation, training targets and inference procedure are defined in Sec. 3. The appendix details data preparation (Appendix A), implementation and training (Appendix B), and evaluation protocols and baseline settings (Appendix C). We will publicly release our inference code and evaluation pipelines, together with code to reproduce the image and video visualizations.

References

Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo. SAM 3: Segment Anything with Concepts. arXiv preprint arXiv:2511.16719.

Yue Chen, Xingyu Chen, Yuxuan Xue, Anpei Chen, Yuliang Xiu, and Gerard Pons-Moll. Human3R: Everyone Everywhere All at Once.

Forrester Cole, Kyle Genova, Avneesh Sud, Daniel Vlasic, and Zhoutong Zhang. Differentiable Surface Rendering via Non-Differentiable Sampling.

1. Improved 4D human motion tracking across articulated clothing and hair by introducing Dense4D, a feed-forward model that predicts Human Point Maps (HPMs) and depth for each image. This allows for amodal correspondence: points hidden in the reference image can be localised from observations in other frames without needing to predict camera poses or large scene displacements.

2. Enhanced human geometry reconstruction by enabling template-free dense 4D human correspondence across the body, clothing and hair from a set of monocular RGB images. This system assembles observations of the body, clothing and hair into a coherent reference-time geometry, providing visible evidence of consistent dense correspondence through articulation and clothing deformation.

3. Decoupled scene understanding from human tracking by allowing direct matching across long temporal intervals. The paper states, Dense4D supports direct matching across long temporal intervals and integrates with scene reconstruction through shared image pixels, which enables the system to reconstruct recognisable bodies yet place transferred points away from their own surfaces in comparisons against baselines.

4. Precise 3D localization of observations using an amodal 3D localisation protocol on 4D-DRESS. This protocol measures how closely observations across time align with the full reference-time scan, achieving a mean one-sided point-to-scan distance of 0.86 cm, which is significantly better than baselines like V-DPM (2.31 cm).

5. Robust trajectory prediction for real people by ensuring consistency between correspondence and geometry. Dense4D ensures that tracks and per-frame depth are consistent by construction because Each 3D position uses the depth predicted at the matched target pixel, leading to more precise trajectories compared to baselines like V-DPM.

6. Improved robustness against occlusions in dynamic scenes by utilizing a confidence-weighted objective function. The training minimizes a loss where Lk = Q.98[rki σki − 0.2 log σki]i + Q.98[rki]i, which effectively handles uncertainty, leading to higher occlusion accuracy (OA) in tracking benchmarks.

7. Improved input conditioning for crop-aware prediction by incorporating a crop embedding that conditions the aggregator on cropping geometry. This allows the model to utilize information such as focal length and zoom, which is shown to improve reconstruction error on 111 of 128 test tracks when compared to an unconditioned variant.