OmniEgoCap: Device-Agnostic Sequence-Level Egocentric Motion Reconstruction

arXiv:2512.19283 · cs.CV · Submitted 2025-12-22 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "OmniEgoCap: Device-Agnostic Sequence-Level Egocentric Motion Reconstruction".

Jane: The gist:

Tom: First, who's behind it and why it matters.

Paper summary: Lu: So looking at the title, "OmniEgoCap: Device-Agnostic Sequence-Level Egocentric Motion Reconstruction," what it really means for the field is that they’ve moved from making solutions for a single camera to building a generalized system.

Jane: It means we are getting closer to systems that can truly understand three dee human movement across all the cameras people use, not just one specific type of sensor or phone <ref:2512.19283#pg1>.

Tom: They achieved this by shifting the focus from short-term estimation to sequence-level inference combined with geometry augmentation, which solves the issue of head trajectory ambiguity and handles intermittent hand observations better.

Meng: From my point of view, it means a more robust system for applications that need to track movement over long periods, like analyzing how people move in a room or understanding complex interactions.

Lalam: The work establishes state-of-the-art performance on public benchmarks and demonstrates robust cross-device generalization without needing hardware-specific retraining.

Jane: It’s about providing physical anchors through consistent body shape and sequence level context to resolve the ambiguity caused by sparse visual data.

Tom: While they noted that the optimization is mainly for offline processing, this paper lays a solid foundation for understanding how to build more versatile AI tools for egocentric vision.

Conclusion: Tom: So, to recap: OmniEgoCap is about taking that egocentric video—the one from your phone or camera—and reconstructing the person's full body motion without needing to retrain for every new camera setup it sees.

Jane: That’s the main idea, right? It tackles this big problem where you can’t just use a model trained on one phone to work perfectly on another.

Lu: The authors are looking at head movements and hand movements which are tricky because they only show what the camera sees at that exact second.

Meng: From a deployment standpoint, what I see here is that they’re trying to build something that doesn't require custom calibration for every single device you use in the wild.

Lalam: This paper shows a sequence-level diffusion framework, meaning it looks at the whole video as one long thing instead of just taking short chunks to estimate motion.

Tom: That sequence level approach is key because it lets them capture these long range physical constraints, like how tall the person is or their general body shape over a whole walk.

Jane: So they’re using those longer stretches of time to figure out the basic physics of movement, which then helps them guess what's happening in the short, blurry moments where the hands disappear.

Lu: They introduce geometry aware visibility augmentation to trick the model into seeing things it wouldn't normally see on a specific lens or mount.

Meng: That sounds smart for generalization because it’s not just relying on one specific camera angle; they’re teaching it about how different cameras distort reality.

Lalam: The results show superior reconstruction across synthetic, real devices, and even in the wild settings where things are messy.

Tom: It means this isn't just a lab trick; it’s showing robust performance when you actually put these models in use outside of a controlled test room.

Jane: What this really changes for us is that future egocentric applications, like augmented reality or fitness tracking, can rely on one general model instead of needing hundreds of specialized ones.

Lu: We still have to consider the latency for real time use, though they mention it’s currently optimized for offline processing.

Meng: That sounds like a necessary limitation right now; if it runs too slow on a phone, it’s not useful for immediate feedback loops.

Lalam: The core message is that by focusing on sequence level context and physical invariants like body shape, you can get consistent motion even when the input data—the specific camera view—is wildly different.

Tom: So we’ve moved past just making a model that works for one setup to building something that adapts to many setups.

Jane: It really shows how using diffusion models with these kinds of physical constraints can solve the messy problem of reconstructing human motion from imperfect, intermittent video feeds.

Seoul National University

cs.CV

Submitted: 2025-12-22

Updated: 2026-10-19

Comments: Accepted to NeurIPS 2026. Project Page: https://kyungwoncho.github.io/OmniEgoCap/

Project page: https://kyungwoncho.github.io/OmniEgoCap

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: The gist: OmniEgoCap presents a unified diffusion framework that scales egocentric reconstruction to diverse capture setups by shifting from short-term windowed estimation to sequence-level inference

Key concepts

Sequence-level Diffusion Framework: Instead of estimating motion in small, isolated time windows, this method processes the entire video sequence at once using a diffusion model. This captures long-range physical relationships and global context about the wearer and camera visibility, leading to more reliable 3D reconstruction.
Geometry-aware Visibility Augmentation: This strategy simulates real-world camera variations during training by artificially creating diverse occlusion patterns and field-of-view changes. This exposes the model to different lens distortions, tilts, and aspect ratios, making it robust when deployed on unseen devices.
Body Shape Invariants ($eta$ and G): The model learns consistent physical body shapes ($eta$) and global context features (G) from the sequence data. These learned invariants act as strong constraints that help disambiguate ambiguous cues, such as head movements alone, ensuring the reconstructed motion remains physically plausible.
Device-Agnostic Reconstruction: The core goal is to create a system that works reliably on any egocentric device without requiring new training for each camera setup. By focusing on sequence-level evidence and geometry augmentation, the framework generalizes well across diverse mounting configurations and optics.

Terminology

Benchmark — 1× in this paper · explained in 2 episodes
Generalization — 1× in this paper · explained in 13 episodes

Summary

The gist: OmniEgoCap presents a unified diffusion framework that scales egocentric reconstruction to diverse capture setups by shifting from short-term windowed estimation to sequence-level inference and introducing a geometry-aware visibility augmentation strategy, establishing a new state-of-the-art on public benchmarks and demonstrating robust performance across diverse, in-the-wild environments<ref:2512.19283#pg2>

How it works

OmniEgoCap is a sequencelevel diffusion framework that leverages longrange physical invariants and geometryaware augmentation to reconstruct consistent 3D fullbody motion across diverse egocentric camera setups<ref:2512.19283#pg2>. The core idea is to process entire motion sequences instead of short temporal windows, which enables reliable estimation of invariant attributes such as height, body proportions, and camera visibility boundaries inferred from periods of standing or regular walking<ref:2512.19283#pg3>. This approach captures a global perspective and recovers invariant physical attributes that provide critical constraints for disambiguating head-only cues<ref:2512.19283#pg4>.

The framework employs a transformer-based diffusion architecture where the encoder processes spatiotemporally invariant conditioning features to predict a single body shape and perframe summary features, and the diffusion decoder denoises a noisy canonicalized pose to reconstruct the fullbody motion<ref:2512.19283#pg5>. The encoder E produces three subcomponents: shape cues, global context G, and local summaries S1:T<ref:2512.19283#pg7>. The global context G is injected via AdaLN-Zero to enable global modulation of the local aggregation process<ref:2512.19283#pg7>.

Key Components and Strategies

The method introduces several key components to achieve deviceagnostic reconstruction<ref:2512.19283#pg4>. These include a dedicated encoder-decoder architecture where the motion-conditioned encoder and diffusion-based decoder jointly model longrange dynamics and consistent body shape<ref:2512.19283#pg5>. The representation involves converting raw signals into a normalized representation invariant to both spatial and temporal variations, such as defining the perframe canonical frame at the head’s floor projection<ref:2512.19283#pg7>.

A crucial strategy is the geometryaware visibility augmentation strategy, which simulates diverse hand visibility boundaries and occlusion patterns during training<ref:2512.19283#pg4>. This augmentation exposes the model to diverse configurations such as arbitrary field-of-views (FoV), camera tilt, varying aspect ratios, lens masks, and optical distortions<ref:2512.19283#pg8>. To ensure robustness against imperfect in-the-wild tracking, stochastic temporal signal drops and pose perturbations are applied using heavy-tailed distributions like Poisson and Log-Normal to model realworld occlusions<ref:2512.19283#pg8>.

Loss Functions and Training

The training objective L combines three terms: Lsimple, Lshape, and Laux<ref:2512.19283#pg9>. The Lsimple term is the standard DDPM objective for denoising the motion<ref:2512.19283#pg9>. The shape consistency is enforced by minimizing the 3D T-pose joint error, Lshape, which minimizes the 3D T-pose joint error<ref:2512.19283#pg9>. Finally, Laux regularizes the reconstructed motion to align with 3D physical constraints via joint position Lpos and foot skating Lskat losses<ref:2512.19283#pg10>. The auxiliary loss is defined as Laux scaled by alpha n to enforce physical accuracy primarily when the signal level is high<ref:2512.19283#pg10>.

Performance and Generalization

Experiments on public benchmarks and in-the-wild data demonstrate the effectiveness of each component, establishing stateof-the-art performance for egocentric fullbody motion reconstruction<ref:2512.19283#pg4>. The model outperforms baselines across diverse simulated settings, including those with varying FoVs and tilts, showing consistent superiority over 120◦-trained specialists<ref:2512.19283#pg8>. This robust performance stems from the unique structural design rather than just augmented training data alone<ref:2512.19283#pg8>. The framework generalizes remarkably well to unseen, strictly bounded pinhole FoVs by effectively incorporating available visual cues, preventing severe geometry overfitting exhibited by prior diffusion-based models<ref:2512.19283#pg8>.

Ablation Study Insights

Ablation studies confirm the necessity of key design choices; for instance, removing the global context G degrades MPJPE to 83.53mm, proving an identity-preserving shape β is crucial<ref:2512.19283#pg15>. Furthermore, removing Laux (No Auxiliary) causes a 39% Jerk spike and degrades endeffector accuracy, confirming Laux is indispensable for handaware models<ref:2512.19283#pg15>. The No Root Ori variant performs worst because it removes 3DoF, eliminating flexibility by making the root’s orientation entirely dependent on the head<ref:2512.19283#pg15>.

Conclusion and Limitations

OmniEgoCap successfully treats intermittent hand observations as geometric constraints, enabling deviceagnostic reconstruction that generalizes to unseen optics and mounting configurations without hardware-specific retraining<ref:2512.19283#pg6>. The framework provides physical anchors through consistent body shape β and sequencelevel context G, resolving head-trajectory ambiguity<ref:2512.19283#pg6>. However, the current optimization is primarily for offline processing rather than lowlatency, realtime inference<ref:2512.19283#pg6>. Future research should focus on improving robustness to tracking failures and relaxing assumptions about a flat ground plane and detailed finger articulation<ref:2512.19283#pg6>. This establishes OmniEgoCap as a deviceagnostic solution generalizing across diverse devices and configurations in the wild<ref:2512.19283#pg8>.

Acknowledgements

This work was supported by KT (Korea Telecom)<ref:2512.19283#pg16>. The corresponding author is H. Joo<ref:2512.19283#pg16>. The references listed are extensive, covering foundational works in motion capture, diffusion models, and egocentric vision<ref:2512.19283#pg18>.

References

The references cover a wide range of related work including motion capture from body-mounted sensors [6] through physical inertial poser (pip) [79]<ref:2512.19283#pg5>. The paper builds on diffusion models as powerful generative priors for highfidelity motion synthesis [22, 38, 64, 85]<ref:2512.19283#pg5>. Specific related methods include EgoPoser [41], EgoAllo [77], and Avatarposer [42]<ref:2512.19283#pg8>. The work also draws upon techniques from various venues such as CVPR, ICCV, NeurIPS, and ECCV to situate its contributions within the broader field<ref:2512.19283#pg18>.

Supplementary Material Details

The supplementary material provides implementation details for the network architecture, including specific module configurations for the transformer blocks<ref:2512.19283#pg21>. It also details the precise hyperparameters for the geometryaware augmentation strategy, including sampling ranges and constraints on spatial parameters like gamma x and phi y<ref:2512.19283#pg21>. The training process utilizes an Exponential Moving Average (EMA) decay of 0.9999 to stabilize training<ref:2512.19283#pg21>. Inference is accelerated using 30 DDIM steps, and test-time guidance optimization specifically targets the 3D local rotations of the arm joints<ref:2512.19283#pg21>. The calibration procedures for camera-to-head mapping and ground plane fitting are presented as lightweight one time preprocessing procedures<ref:2512.19283#pg24>. The work is supported by KT (Korea Telecom)<ref:2512.19283#pg16>.

Code and Pretrained Checkpoints

Code and pre-trained checkpoints will be released publicly<ref:2512.19283#pg21>. The paper emphasizes that the framework is designed to be deviceagnostic, avoiding hardware-specific retraining<ref:2512.19283#pg6>.

Improvements for AI systems

Bold header: Unified Sequence-Level Diffusion Framework

OmniEgoCap can reconstruct full-body 3D motion across diverse egocentric camera setups by shifting from short-term windowed estimation to sequencelevel inference, which captures a global perspective and recovers invariant physical attributes, such as height and body proportions.

Bold header: Geometry-Aware Visibility Augmentation

The system can robustly handle intermittent hand cues across various devices by introducing a strategy that treat[s] intermittent hand appearances as principled geometric constraints rather than missing data, thereby achieving hardware-agnostic generalization to unseen camera setups.

Bold header: Sequence-Level Context Integration

The improved system can resolve head-trajectory ambiguity by leveraging sequence-level context, as the model predicts a single shape β for a whole sequence and uses global context G to infer that the initially unobserved hands must have been outside the FoV based on later visibility evidence.

Bold header: Physical Consistency via Auxiliary Loss

The system ensures kinematically plausible reconstructions by minimizing position errors, as the loss function includes Laux, which is crucial for satisfying the hand condition H and generating stable and accurate motion, penalizing 3D position errors that cause severe artifacts, such as body penetration.