CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interaction

arXiv:2512.11988 · cs.CV · Submitted 2025-12-12 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interaction".

Jane: Here's the gist of the paper. The gist: CARI4D presents a category agnostic method that reconstructs 4D human-object interaction from monocular RGB video at metric scale and tracks it consistently across frames, outperforming prior art by over 35% in reconstruction error Core Problem and Goal Capturing human-object interaction from RGB cameras is important for human understanding, gaming, and robot learning.

Tom: Reconstruction from RGB alone introduces several challenges, including the variation in shape and pose between subjects or categories, the lack of depth information making it difficult to recover scale and detailed geometry, and the need for robustness to heavy occlusions To address these limitations, CARI4D proposes a method that bridges research and real-world usage by reconstructing objects directly from input without pre-defined models, generalizing beyond fixed categories, and maintaining consistent metric reconstruction and tracking across video frames.

Jane: Method Overview The CARI4D framework is designed to solve the challenging problem of 4D human-object interaction reconstruction from monocular RGB video.

Tom: The process involves several key stages: Coarse-to-fine scale estimation for metric object shape reconstruction: The method assumes the object is mostly visible in the first frame to obtain the object reconstruction and metric scale, utilizing fBRS

39: for mask generation and Hunyuan3D-2

42: for mesh reconstruction.

Title and authors: Jane: So the summary of CARI4D lays out a multi-stage process. First, they do a coarse-to-fine scale estimation to get the metric object shape reconstruction using tools like fBRS for masking and Hunyuanthree dee-two for mesh reconstruction <ref:2512.11988#pg3>.

Lu: They use UniDepth to estimate metric depth and then FoundationPose to figure out the six degrees of freedom pose for those rescaled meshes based on that depth and the camera's intrinsics <ref:2512.11988#pg3>. The scale selection is done by picking the one with minimum chamfer distance, followed by a fine search in a range to find the optimal metric object reconstruction O.

Meng: So they start with an initial guess based on what's visible in the first frame to get that first metric shape and scale. That makes sense for getting started quickly, but what about tracking it over time when things move?

Tom: That’s where the next part kicks in, and it uses Foundation models to initialize the human and object poses in metric space. They use UniDepth again with the metric object mesh to get depth estimation and then FoundationPose per frame for the object poses <ref:2512.11988#pg3>.

Jane: And they added a novel pose hypothesis selection algorithm that uses mask IoU and temporal smoothness to pick the best pose from a list of candidates, which is designed to make those initial pose estimations more robust under occlusion or movement <ref:2512.11988#pg1>.

The paper's summary: Jane: Now let's talk about the specific improvements they claim. They introduce CoCoNet, which is their category agnostic contact reasoning model. It works in a render-and-compare paradigm with spatiotemporal attention to look at joint three dee estimation, input observations, and temporal cues <ref:2512.11988#pg1>.

Tom: So CoCoNet takes the initialized poses and predicts updated poses and binary contact labels indicating if each hand is touching the object by looking at renderings of the human and object under those initial poses along with the input observations <ref:2512.11988#pg4>.

Lu: They also tackle depth alignment when training CoCoNet by aligning estimated depth to ground truth depth using medians of predicted and ground truth depths, which removes absolute translation error from the depth estimations so the network can focus on reasoning about the relative poses between the human and object <ref:2512.11988#pg4>.

Meng: That sounds like a smart way to train it. If you remove that absolute translation error, you're forcing it to learn what's happening relative to each other, which should make the contact reasoning much more reliable for deployment <ref:2512.11988#pg4>.

Lalam: Finally, they use a contact-aware joint optimization framework for the final refinement step. They use a loss function that includes a contact loss Lc, which penalizes large distances between hand and object when contacts are predicted, defined as P i d(J h i, O′ i) * ci <ref:2512.11988#pg4>.

Jane: They also have losses for 2D joint projection, object mask, penetration, and acceleration to make sure the final result looks physically realistic and doesn't have weird overlaps or impossible movements <ref:2512.11988#pg4>.

The paper's improvements: Tom: So to wrap up on CARI4D: it’s about category agnostic 4D reconstruction from monocular RGB video at metric scale, which they claim improves the Chamfer distance by over thirty-five percent for both in-distribution and unseen datasets <ref:2512.11988#pg1>.

Jane: The biggest implication I see is that it moves us away from needing specific object templates, which means we can apply this to those wild videos on the internet that we couldn't train a separate model for before, which they show works even on the InterCap dataset <ref:2512.11988#pg4>.

Lu: It’s powerful because it generalizes well to in-the-wild videos where object categories are completely unseen, demonstrating this category agnostic capability <ref:2512.11988#pg4>.

Meng: From an engineering standpoint, the runtime is fast; they process one video of three hundred frames in about forty-five minutes, which is much quicker than some other methods we've seen <ref:2512.11988#pg4>. But I do have to point out their limitation: they don't explicitly regress detailed finger articulations, which is something you’d need for super fine tasks like grasping plates <ref:2512.11988#pg6>.

Lalam: They also noted a bottleneck where if the initializer fails significantly, like FoundationPose predicts a flipped one hundred eighty-degree pose under rapid motion or heavy occlusion, that can cause issues <ref:2512.11988#pg6>.

Tom: So it’s a solid method for getting metric human-object interaction reconstruction from video without pre-defined models, even if we need to watch out for those initialization hiccups. That covers what they did with CARI4D.

Conclusion: Tom: So we’ve been digging into CARI4D, which is about category agnostic 4D reconstruction of human-object interaction from monocular RGB video, and basically, they claim over a thirty-five percent improvement in reconstruction error compared to what was out there before <ref:2512.11988#pg1,category agnostic 4D reconstruction of human-object interaction>.

Jane: That’s a big number because it means this method can reconstruct these 4D interactions—the shape, the pose, and how the human and object are touching each other—using just one regular RGB video, without needing any prior knowledge of what kind of object or person it is <ref:2512.11988#pg1>.

Lu: The core idea is using a coarse-to-fine scale estimation strategy to get that metric shape reconstruction from the first frame, then using FoundationPose for initial human and object pose estimations in metric space.

Meng: From a deployment standpoint, getting those initial poses right under heavy occlusion is tricky, so they used this novel pose hypothesis selection algorithm that filters candidates based on mask intersection over union and temporal smoothness.

Lalam: And then they use CoCoNet to refine those initial guesses by predicting updated poses and contact labels using a render-and-compare paradigm with spatiotemporal attention.

Tom: That refinement step is where the real meat is, right? They tackle depth alignment during training by aligning estimated depth to ground truth depth using medians of predicted and ground truth depths, which lets the network focus on the relative poses between them.

Jane: I wonder what this means for things like robotics or augmented reality, because if you can consistently track these metric poses across frames without needing a perfect camera setup or a three dee model to start with, that opens up so much room for real-time interaction systems.

Lu: It really pushes the boundary on category agnostic reconstruction. They show it works on in-the-wild internet videos, which is crucial because most models are trained on specific datasets and struggle when they see something new.

Meng: But I gotta say, the limitation they mentioned is that they don't explicitly regress detailed finger articulations, which means if you need to grasp something with extreme precision, this model might not give you the fine detail needed for that task.

Tom: True, and there’s that dependency bottleneck with FoundationPose; if it gets a flipped pose under really fast motion or heavy occlusion, the whole thing can go sideways.

Jane: It sounds like they are pushing towards a system where the AI doesn't just sees an object, but actually understands the physical relationship—the contact and the metric scale—regardless of what that object is.

Lu: They also showed that ground truth object shape or depth information can actually boost reconstruction accuracy when used as privileged information, which is a useful piece of evidence for how to improve these kinds of models.

Lalam: So overall, CARI4D moves us towards more robust 4D human-object interaction understanding from just a single video input, and they’ve made the code and the pretrained models public so we can test this out <ref:2512.11988#pg1>.

Tom: Yeah, it’s a solid piece of work that shows how you can build a system that is both fast enough for production and accurate enough to matter.

Jane: It definitely makes me think about what kind of interactive scenes we could build if we had this level of reliable 4D tracking available everywhere <ref:2512.11988#pg1>.

Xianghui Xie, Bowen Wen, B Yan Chang, Hesam Rabeti, Jiefeng Li, Ye Yuan, Gerard Pons-Moll

NVIDIA University of Tubingen AI Center Institute for Informatics

cs.CV

Submitted: 2025-12-12

Updated: 2026-10-19

Comments: Update table 1 and table 2 after fixing a bug in evaluation

Code: https://github.com/cmu-perceptual-computinglab/openpose

Project page: https://nvlabs.github.io/CARI4D

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 87/100

The gist: The gist: CARI4D presents a category agnostic method that reconstructs 4D human-object interaction from monocular RGB video at metric scale and tracks it consistently across frames, outperforming

Key concepts

Metric Scale Reconstruction: This refers to reconstructing the human and object geometry at real-world physical sizes. CARI4D achieves this by estimating depth and using scale selection methods, ensuring the resulting 3D meshes accurately represent the actual dimensions of the interaction, moving beyond arbitrary model sizes.
CoCoNet: This is a category-agnostic contact reasoning model that uses a render-and-compare approach. It takes initial pose estimations and input images to predict updated poses and binary labels indicating where hands are touching objects, allowing the system to refine interactions dynamically.
Category Agnostic: The method is designed to work with any object category without needing specific training data for that object type. This generalization is achieved by leveraging foundation models and focusing on metric-scale geometry and interaction reasoning rather than fixed object shapes.
Contact-Aware Joint Optimization: This final refinement step uses a complex loss function to ensure the predicted human and object poses are physically realistic. It includes losses for hand distance, 2D joint projection, penetration depth, and acceleration, forcing the model to generate smooth motions with accurate physical contact.

Terminology

Foundation Model — 1× in this paper · explained in 3 episodes
Foundation Models — 1× in this paper · explained in 7 episodes
Physical Constraints — 1× in this paper · explained in 7 episodes
Reconstruction Error — 1× in this paper · explained in 2 episodes

Summary

The gist: CARI4D presents a category agnostic method that reconstructs 4D human-object interaction from monocular RGB video at metric scale and tracks it consistently across frames, outperforming prior art by over 35% in reconstruction error

Core Problem and Goal

Capturing human-object interaction from RGB cameras is important for human understanding, gaming, and robot learning. Reconstruction from RGB alone introduces several challenges, including the variation in shape and pose between subjects or categories, the lack of depth information making it difficult to recover scale and detailed geometry, and the need for robustness to heavy occlusions To address these limitations, CARI4D proposes a method that bridges research and real-world usage by reconstructing objects directly from input without pre-defined models, generalizing beyond fixed categories, and maintaining consistent metric reconstruction and tracking across video frames.

Method Overview

The CARI4D framework is designed to solve the challenging problem of 4D human-object interaction reconstruction from monocular RGB video. The process involves several key stages:

** Coarse-to-fine scale estimation for metric object shape reconstruction:**

The method assumes the object is mostly visible in the first frame to obtain the object reconstruction and metric scale, utilizing fBRS [39] for mask generation and Hunyuan3D-2 [42] for mesh reconstruction. Metric depth is estimated using UniDepth [32, 33], and FoundationPose [52] is used to estimate the 6DoF pose for rescaled meshes based on depth and intrinsics. The scale with minimum chamfer distance is selected, followed by a fine-grained search within a range of scales to select the optimal metric object reconstruction O.

Human and Object Pose Initialization:

Foundation models are leveraged to obtain robust initialization of human and object pose estimations that are aligned in metric-scale space. This involves using the metric-scale object mesh to perform depth estimation with UniDepth [33] and solving per-frame object poses via FoundationPose [52]. A novel pose hypothesis selection algorithm is introduced to improve FoundationPose results by considering mask IoU and temporal smoothness, selecting the best pose from a list of candidates. For human estimation, NLF [37] is used per-frame, and its predictions are aligned with the estimated metric depth via an iterative closest point manner.

Interaction Refinement and Contact Reasoning

To refine the initial estimations and reason about contacts, CARI4D introduces CoCoNet, a Category agnostic Contact reasoning model. This model operates in a render-and-compare paradigm with spatiotemporal attention to leverage joint 3D estimation, input observations, and temporal cues.

CoCoNet Function:

Given initialized poses from Sec. 3.2, CoCoNet predicts updated poses and binary contact labels indicating if each hand is in contact with the object. The network consumes renderings of the human and object under their initialized poses, along with input observations, to predict delta updates for pose parameters and hand contacts formally as fϕ: (Iˆ(Hˆi, Oˆi), I) 7→

∆Hˆ i, ∆Oˆ i, ci L.

Depth Alignment:

When training CoCoNet, a depth alignment strategy is proposed to align estimated depth to ground truth depth based on medians of predicted and GT depths. This alignment removes absolute translation error from depth estimations, allowing the network to focus on reasoning the relative poses between human and object.

Contact-based Joint Optimization

The final refinement step uses contact-aware joint optimization to ensure contact realism and motion smoothness. The objective function is defined as L = λcLc + λj2dLj2d + λmLm + λpenLpen + λaccLacc.

Loss Components:

The contact loss Lc penalizes large distances between hand and object when contacts are predicted, defined as Lc = P i d(J h i, O′ i) · ci. The 2D joint projection loss Lj2d minimizes the distance between projected 2D joints and detected 2D joints using OpenPose [1]. The penetration loss Lpen uses Volumetric SMPL [27] to predict a signed distance, defined as Lpen = P i P q∈O′ i ReLU(−ΦSMPL(q)) squared.

Experimental Results and Generalization

Experiments on the BEHAVE dataset show that CARI4D significantly outperforms baselines, achieving over 35% improvement in Chamfer distance for both in-distribution and unseen datasets. The method generalizes well to in-the-wild videos where object categories are completely unseen, demonstrating category agnostic capability. Ablation studies confirm the effectiveness of the proposed modules, including the dynamic pose selection algorithm and CoCoNet. Furthermore, testing on in-the-wild internet videos shows results consistent with those on BEHAVE.

Limitations and Future Work

A limitation noted is that the method primarily targets full-body human-object interaction and does not explicitly regress detailed finger articulations, which can be important for fine-grained manipulation tasks like grasping plates. Another dependency bottleneck arises when the initializer fails significantly, as FoundationPose can occasionally predict flipped 180-degree poses under rapid motion or severe occlusion. Future work could integrate specialized hand pose estimators to solve for hand parameters separately and fuse them with the full-body kinematic chain via optimization. The paper also shows that ground truth object shape or depth improves slightly the object reconstruction when privileged information is given.

Runtime Performance

The method achieves a fast runtime while producing the most accurate results, outperforming PICO and InterTrack. The average runtime to process one video of 300 frames is 45 minutes for CARI4D compared to 1560 minutes for PICO.

Oracle Study

The performance of the base model designed for RGB input is close to the upper bound when ground truth object shape or depth information is provided. The combination of estimated depth and ground truth depth achieves the most boost in reconstruction accuracy.

Conclusion

CARI4D presents a category agnostic method to reconstruct 4D human object interaction from monocular RGB video by aligning foundation model predictions, introducing learned contact reasoning via CoCoNet, and performing contact-aware joint optimization. CARI4D achieves over 35% improvement in Chamfer distance for both in-distribution and zero-shot settings, demonstrating strong generalization to diverse object shapes and interactions. The code and pretrained models are released publicly.

--- Page 6 ---

Figure 7. CoCoNet architecture. Here b, t, h, w denote batch size, temporal window size, image height and width respectively

--- Page 7 ---

Figure 2. CARI4D method overview. Given a monocular RGB video, we reconstruct the 4D human and object at metric scale with consistent contacts

--- Page 3 ---

Figure 3. Qualitative comparison on BEHAVE dataset [3]. InterTrack [59] reconstructs human and object as point clouds only, and the shapes are noisy

--- Page 4 ---

Figure 4. Zero-shot generalization to unseen InterCap dataset [18].

Improvements for AI systems

The system can perform category-agnostic 4D reconstruction of human-object interactions from monocular RGB videos at metric scale, tracking poses and contacts consistently across the video as stated in Figure 1 caption.
The AI system can generalize zero-shot to in-the-wild internet videos by applying its method to novel object categories not seen during training, as demonstrated by achieving results on the InterCap dataset.
The system can generate highly accurate, metric-scale 4D reconstructions of human and object meshes from a single RGB video input using a coarse-to-fine scale estimation strategy and FoundationPose for initial pose estimation.
The AI system can robustly track object pose under heavy occlusion by employing a pose hypothesis selection algorithm that robustly tracks object pose under heavy occlusion, filtering candidates based on mask IoU and temporal smoothness.
The system can refine initial human and object poses to ensure physical consistency by using the CoCoNet model, which predicts updated poses and binary contact labels by rendering estimations into RGB/depth/masks, leveraging a render-and-compare paradigm.
The AI system can guarantee contact realism and avoid penetration errors through a contact-aware joint optimization framework that minimizes a loss function including the contact loss Lc, alongside losses for 2D joint reprojection, object mask, penetration, and acceleration.
The system can handle complex interaction scenarios more accurately by incorporating oracle study information; specifically, ground truth object shape or depth can improve reconstruction accuracy when used as privileged information.
The system can be made more robust to initialization errors by aligning the initial human and object poses to ground truth depth using a transformation derived from medians of predicted and GT depths, ensuring the network focuses on reasoning the relative poses between human and object.

Abstract

Accurate capture of human-object interaction from ubiquitous sensors like RGB cameras is important for applications in human understanding, gaming, and robot learning. However, inferring 4D interactions from a single RGB view is highly challenging due to the unknown object and human information, depth ambiguity, occlusion, and complex motion, which hinder consistent 3D and temporal reconstruction. Previous methods simplify the setup by assuming ground truth object template or constraining to a limited set of object categories. We present CARI4D, the first category-agnostic method that reconstructs spatially and temporarily consistent 4D human-object interaction at metric scale from monocular RGB videos. To this end, we propose a pose hypothesis selection algorithm that robustly integrates the individual predictions from foundation models, jointly refine them through a learned render-and-compare paradigm to ensure spatial, temporal and pixel alignment, and finally reasoning about intricate contacts for further refinement satisfying physical constraints. Experiments show that our method outperforms prior art by 38% on in-distribution dataset and 36% on unseen dataset in terms of reconstruction error. Our model generalizes beyond the training categories and thus can be applied zero-shot to in-the-wild internet videos. Our code and pretrained models will be publicly released.