What 30,000 Hours of Ego-centric Video Does Not Teach

arXiv:2610.12464 · cs.CV · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "What 30,000 Hours of Ego-centric Video Does Not Teach".

Jane: The gist World models offer a promising alternative to physics-based simulators, yet remain far from practical deployment;

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper from arXiv called "What thirty thousand Hours of Ego-centric Video Does Not Teach," and the authors are Jiahua Dong, Anurag Bagchi, Yash Jangir, Muhammad Zubair Irshad, and Sergey Zakharov <ref:2610.12464#pg1,What 30,000 Hours of Ego-centric Video Does Not Teach>. What’s the big takeaway here for us?

Jane: It’s about how far we can scale these ego-centric human video models before they actually start predicting how things in the world will react to an agent’s actions. Basically, scaling up the data doesn't automatically make the model good at simulating physics or object interactions.

Lu: The paper is questioning whether just feeding an agent a massive amount of video showing it interacting with objects is enough for those models to actually understand how those objects behave when the agent moves them. It points out a gap between modeling what the agent *does* and modeling what happens in the world as a result.

Meng: So, if we look at the scaling data they show, it looks like improving how well an agent models its own body gets pretty good pretty fast, but that doesn't translate to better object fidelity or prediction of world dynamics.

Tom: Exactly. The paper points out that while agent fidelity climbs steadily toward usable levels with more hours—say 10k hours shows a gain of zero point six SCS between 3k and 10k, but then it slows down way off after that—the object fidelity lags far behind and doesn't show that same deceleration <ref:2610.12464#pg1>.

Jane: And they specifically mention that the agent fidelity saturates at a value of zero point eight one one, but the object limit remains undetermined within their range because it just keeps going without a curve to constrain it <ref:2610.12464#pg1>. That’s a big limitation they point out for scaling ego-centric data.

Lu: They introduce this idea called "object-centric adaptive noise scheduling" as an intervention that tries to fix that specific problem, suggesting we should shift the training signal away from general scene appearance toward the actual dynamics of the objects being manipulated.

Meng: That sounds like a more focused way to train things. How does this noise scheduling actually work in practice? Is it just randomly increasing noise everywhere, or is there something smarter happening?

Jane: It involves localizing dynamic regions using a map called Mdyn, where high scores mean the area is both moving in the world and near the agent’s hands. Then they apply the supervision by either raising the noise level or reweighting the loss specifically within those dynamic areas.

Title and authors: Tom: So it’s about telling the model, "Hey, pay extra attention to what's happening right where you are touching something," rather than just letting it learn from the whole scene appearance. That’s a key shift in focus there.

Lu: The results show that this method exceeds the standard skeleton baseline at every training budget, reaching zero point five four six against zero point five two seven at 30k hours, and it fits that same saturating form but with a higher asymptote than the baseline does <ref:2610.12464#pg1>.

Jane: And they’re not just looking at agent fidelity; they are also showing that these gains transfer to a downstream humanoid setting, meaning this training helps when you try to model how a human interacts with the world in general.

Meng: That’s interesting because modeling the humanoid interaction with the world is still being treated as a bottleneck even after this kind of pre-training. The paper says hand SCS for their method stays nearly flat, falling from zero point eight nine at frame two to zero point eight five at frame sixteen while object SCS falls more than twice as far, from zero point eight two down to zero point seven one.

Tom: So the model knows where the hands will be in space, but it still struggles to correctly predict what happens in the world when those actions occur, which is exactly what we need for real-world use cases.

Jane: The authors themselves flag that they can't get rid of irreducible uncertainty regarding physical properties like mass or friction, and that the quantities that matter most—like contact forces—aren't observable in pixels at all.

Lu: They argue that this is precisely where a physics simulator fits in because it can supply supervision for those unobservable things, which makes the current limitation of ego-centric data more apparent.

Meng: So they’re suggesting that relying solely on video might be incomplete because the necessary physical details are missing from the pixel data itself.

Tom: Right. And looking at the model design, it predicts future frames given an initial RGB observation xzero a text description c, and an agent-motion sequence a0:T. It uses a backbone that processes text and action tokens jointly with video tokens.

Jane: They train this on a balanced ladder of data spanning three hundred to 30k hours, with the hours measuring the fraction of the corpus consumed relative to its total size of thirty thousand hours <ref:2610.12464#pg1>. It’s a lot of data they are working with.

Title and authors: Lu: They also test an ablation study where they check if always supplying a projected skeleton limits object interaction learning by reducing the model’s reliance on visual inference. They compare Cosmos three Nano with standard skeleton conditioning against a variant trained with skeleton dropout, and both receive the full sequence during evaluation <ref:2610.12464#pg1>.

Meng: The random-region baseline confirms that the benefit of dynamic region supervision comes from where supervision is shifted rather than just from adding more noise or perturbation to the data itself.

Tom: It seems like they’re trying to isolate exactly which part of the training process is responsible for improving object interaction, and it’s not just brute force data scaling.

Jane: And when you look at the humanoid fine-tuning part, they show that their design contributes more than twice as much on top of ego-centric human pre-training alone when fine-tuned on ground truth demonstrations from eleven tasks from the simulated bimanual benchmark.

Lu: They note a specific departure here: the gains are larger for the agent itself than for the objects it manipulates, which is something they observe in their humanoid fine-tuning results.

Meng: So, even when we move to downstream humanoid tasks, this pre-training helps more than just standard ego-centric training alone does.

Tom: To wrap this up, the paper "What thirty thousand Hours of Ego-centric Video Does Not Teach" is really telling us that scaling up human interaction data has hit a wall when it comes to accurately modeling the physics and dynamics of what the agent is doing to those objects <ref:2610.12464#pg1,What 30,000 Hours of Ego-centric Video Does Not Teach>.

Jane: The main thing they are pushing forward is object-centric supervision as a way to focus training on the things that matter most for world simulation, rather than just general scene appearance.

Lu: So, we have an idea now for how we can try to get closer to simulating physics by making the AI focus its attention more precisely on the dynamic parts of the scene.

Meng: From an engineering standpoint, it’s promising because it gives us a targeted way to improve fidelity without needing infinite amounts of data for every single interaction type.

Tom: It means we need better ways to inject physical knowledge into these large models so they don't just become great at mimicking movement but also great at predicting consequences.

Jane: So, the implication is that future progress in this area won't come from just more video hours, but from smarter supervision signals that prioritize object dynamics over general scene visuals.

The paper's summary: Tom: So we're talking about "What thirty thousand Hours of Ego-centric Video Does Not Teach." The big takeaway is that just feeding these models more and more videos of an agent interacting with objects isn't the only thing that makes them smarter.

Jane: Right. They’re saying that scaling up the data for ego-centric video improves how well the agent models its own body, but it doesn't improve how accurately those agents predict what will happen to objects in the world when they move them.

Lu: The paper shows a clear gap between agent fidelity—how well the model understands where your hands are—and object fidelity—how well it predicts where those objects will end up or how they’ll behave.

Meng: So, if I’m building something that needs to interact with the physical world, this suggests that just having endless video data isn't enough for simulation accuracy.

Tom: Exactly. They show that agent fidelity hits a ceiling pretty fast, while object fidelity just keeps climbing without a clear stop sign in their tests.

Jane: And they introduce this idea of object-centric adaptive noise scheduling as the way to fix that gap by shifting the training focus away from general scene looks toward the actual dynamics of what’s being manipulated.

Lu: This method uses a dynamic region map, Mdyn, which identifies areas where things are both moving and near the agent's hands. Then they use that map to adjust the noise level or reweight the loss right there.

Meng: That sounds like a targeted way to train instead of just drowning the model in more raw data where it doesn't matter for object interaction.

Tom: And they prove it works, showing that this approach beats a standard skeleton baseline at every training budget, even out to 30k hours <ref:2610.12464#pg1>.

Jane: They also show this pre-training helps when you try to fine-tune the model for actual humanoid tasks, meaning it transfers those gains over to more general world interaction.

Lu: But they also point out that even with this, the model still struggles with predicting the consequences of actions on objects in a realistic way.

Tom: So, scaling data gets you good agent movements and some object recognition, but you still haven't got that reliable physical prediction yet.

Jane: The authors are really highlighting that we can’t just rely on collecting more ego-centric video to solve this problem alone.

Lu: They suggest the real missing piece is modeling the unobservable things, like friction or stiffness, which is exactly what a physics simulator can provide as supervision.

Meng: That makes sense from an engineering standpoint; if we can get that physical knowledge into the training loop, it might bridge that gap between video and true world simulation.

Tom: So the big point here is that future progress needs to focus less on just more hours of video and more on smarter supervision signals for object dynamics.

The paper's improvements: Tom: So we're talking about how they suggest fixing that object fidelity problem in "What thirty thousand Hours of Ego-centric Video Does Not Teach." They aren't just pointing out a problem; they’re proposing a specific training fix.

Jane: Right. The main idea is to stop looking at the whole scene appearance and instead focus the training on what actually matters: how objects move when you touch them.

Lu: They introduce object-centric adaptive noise scheduling, which is this new way to schedule supervision that targets the dynamics of manipulated objects specifically.

Meng: So instead of raising noise everywhere, they create a dynamic region map, Mdyn, based on three dee displacement and tracking reliability around where the agent’s hands are.

Tom: That map lets them know exactly where the object interaction is happening so they can raise the noise level or reweight the loss only in those spots.

Jane: It’s like telling a student, "Focus all your effort on this specific textbook chapter while you're reading." It changes where they spend their learning time.

Lu: The results show this method consistently beats the standard skeleton baseline at every budget, reaching zero point five four six against zero point five two seven at 30k hours and maintaining that high performance ceiling better than the baseline does <ref:2610.12464#pg1>.

Meng: That's solid because it means we don't need to just throw more data at the model; we can change the signal itself to make it focus better on physics-like behaviors.

Tom: They also showed this helps when you take that pre-trained agent and fine-tune it for actual humanoid tasks, showing how much that ego-centric training actually carries over.

Jane: It proves that the knowledge gained from watching a human move things is useful even when you’re trying to model a full humanoid system.

Lu: But they also pointed out their own limitations, specifically that they can't capture the true physics of things like mass or friction, which is why a real physics simulator would be the ultimate supervisor.

Meng: So what this means for us in practice is that while video training gets you close to movement, we still need to integrate some kind of physical model to get true world interaction accuracy.

Tom: It's a big hint that the next step isn't just bigger datasets; it’s about making our supervision signals smarter and more physically grounded.

Conclusion: Tom: So we're wrapping up "What thirty thousand Hours of Ego-centric Video Does Not Teach." The main point is that scaling up video data doesn't magically solve the problem of simulating how objects react to movement.

Jane: Exactly. They show that agent fidelity improves quickly, but object fidelity lags behind and doesn't follow the same curve, which is a major limitation in current approaches.

Lu: The paper suggests using object-centric adaptive noise scheduling as a fix by focusing the training on dynamic regions defined by a map called Mdyn to improve those object interactions specifically.

Meng: That targeted focus sounds like something we could actually implement to improve simulation accuracy without needing an infinite amount of data for every single scenario.

Tom: It's smart because it shifts where the AI spends its effort, moving it from general scene recognition to precise physical interaction modeling.

Jane: And they showed that this approach transfers well into humanoid tasks, meaning these training methods are useful for broader applications in motion prediction and control.

Lu: But they’re honest about the hard limits too, saying the model still can't capture things like mass or friction perfectly because those details aren't in the pixels.

Meng: That confirms what we suspected; video alone is missing that physical knowledge that a simulator provides for free.

Tom: So, even with this new object-centric supervision, we still need to find a way to inject that actual physics knowledge into the model structure itself.

Jane: It gives us a clearer direction for future research: we need better ways to combine what we see with what we know about the physical world.

Lu: I think this opens up some wild possibilities for how AI can start modeling causality in these visual scenes instead of just pattern matching movement.

Meng: From an engineering standpoint, it means our next round of work should look at how to formally integrate those dynamic region maps into the training pipeline efficiently.

Tom: Well said. We’re definitely leaving this paper with a clear path forward: smarter supervision over sheer data volume for complex world modeling.

Jiahua Dong, Anurag Bagchi, Yash Jangir, Muhammad Zubair Irshad, Sergey Zakharov, Martial Hebert, Homanga Bharadhwaj, Yu-Xiong Wang, Vitor Campagnolo Guizilini, Pavel Tokmakov

University of Illinois Urbana-Champaign · Carnegie Mellon University · Johns Hopkins University · Toyota Research Institute

cs.CV

Submitted: 2026-10-08

Updated: 2026-10-08

Project page: https://fidelity-gap.github.io

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 67/100

The gist: The gist World models offer a promising alternative to physics-based simulators, yet remain far from practical deployment; scaling ego-centric human video brings agent modeling close to its limit

Key concepts

Agent Fidelity
This measures how accurately the model predicts where an agent's body will be in future frames. Scaling ego-centric data improves this fidelity steadily, but it saturates at a certain level, indicating a limit to how much better the agent prediction can get from more data alone.
Object Fidelity
This measures how accurately the model predicts where manipulated objects will be. Unlike agent fidelity, object fidelity does not show clear signs of slowing down as more data is added, suggesting its ultimate achievable level is currently unknown within the tested range.
Object-Centric Adaptive Noise Scheduling
This technique shifts supervision away from general scene appearance to the dynamics of manipulated objects. It works by localizing dynamic regions near the agent's hands and then adjusting noise levels or loss weighting specifically in those areas, which is shown to significantly boost object fidelity.
Structural Consistency (SCS)
This metric evaluates the structural accuracy of rendered agents and objects being interacted with. It checks if the predicted shapes and positions of the agent's hands and manipulated items match reality, providing a measure of how well the model captures physical structure.

Terminology

Summary

The gist World models offer a promising alternative to physics-based simulators, yet remain far from practical deployment; scaling ego-centric human video brings agent modeling close to its limit while leaving its effects on the world far behind; and closing this gap will depend on how models are trained, not only on how much data they see

Scaling Ego-Centric Data Limitations

Scaling ego-centric data improves both the agent and its action effects, but not equally As shown in Figure 1, agent fidelity climbs steadily toward usable levels while object fidelity lags far behind, improving more slowly and from a lower base Better conditioning substitutes for most of the agent gains: 1k hours of skeleton-conditioned training exceed the agent fidelity that standard conditioning reaches at 30k, and both converge to nearly the same limit Agent fidelity is visibly saturating, gaining 0.12 SCS between 300 and 3k hours but only 0.06 between 3k and 30k, placing its asymptote at a value of 0.811 Object fidelity shows no comparable deceleration, and a curve without curvature does not constrain its asymptote, so the object limit remains undetermined within our range

Object-Centric Supervision

We introduce object-centric adaptive noise scheduling (Section 5), which shifts supervision away from overall scene appearance and toward the dynamics of manipulated objects, and is the only intervention we test that raises the level object fidelity converges to This method involves localizing dynamic regions using a map Mdyn, where high scores indicate where something is both moving in the world and near the agent’s hands The supervision is applied by either raising the noise level or reweighting the loss within these dynamic regions Results show that our method exceeds the skeleton baseline at every budget, reaching 0.546 against 0.527 at 30k hours, and fitting the same saturating form places its asymptote above the baseline’s

Transfer to Humanoid Modeling

We show that these gains transfer to a downstream humanoid setting, where our design contributes more than ego-centric pre-training alone does, but modeling humanoid’s interactions with the world remains a bottleneck Our results indicate that hand SCS for OURS is nearly flat, falling from 0.89 at frame 2 to 0.85 at frame 16, while object SCS falls more than twice as far, from 0.82 to 0.71 The model knows where the hands will be, but fails to correctly predict the consequences of agent’s actions in the world

Limitations and Future Directions

The results are not an argument against collecting ego-centric human video; more data improved both axes at every scale we measured, and pre-training transferred to humanoids Two factors plausibly contribute to the remaining gap: irreducible uncertainty regarding physical properties like mass, friction, and stiffness, and that the quantities that matter most are not observable in pixels at all Contacts and forces are never visible, which is precisely what a physics simulator represents exactly and can supply as supervision The ceilings we report are in this sense optimistic because they hold only insofar as the fitted regime continues beyond 30k hours

Model Design and Training

The model predicts future frames given an initial RGB observation x0, a text description c, and an agent-motion sequence a0:T The backbone processes text and action tokens jointly with video tokens, where text attends causally while video and action tokens attend jointly to one another and to the text context We train on a balanced, nested data ladder at 300, 1k, 3k, 10k and 30k hours The training budget is defined such that hours measure the fraction of the corpus consumed relative to its nominal size of 30,000 hours

Evaluation Metrics

We employ structural consistency and perceptual quality metrics, specifically SCS (Bagchi et al., 2026) for agents and objects being interacted with LPIPS (Zhang et al., 2018) serves as a perceptual control, so that differences in structural fidelity can be checked against differences in image quality Our evaluation set comes from a separate recording campaign, and we measure how accurately the agent’s body and the manipulated objects are rendered We report SCS separately for the hands (agent) and the manipulated objects

Ablation Studies

We test whether always supplying the projected skeleton limits object interaction learning by reducing the model’s reliance on visual inference We compare Cosmos 3 Nano with standard skeleton conditioning against a variant trained with skeleton dropout, and both variants receive the full skeleton sequence during evaluation The random-region baseline confirms that the benefit of dynamic region supervision comes from where supervision is shifted rather than from the perturbation itself We find no evidence of a shortcut introduced by skeleton conditioning when comparing checkpoints at 10k training hours

Humanoid Fine-Tuning

We fine-tune Cosmos 3 Nano on ground-truth demonstrations from 11 tasks from the simulated bimanual benchmark of Yang et al. (2025), conditioning on a single start frame and the 138-dimensional humanoid state trajectory Our design contributes more than twice as much on top of it when compared to ego-centric human pre-training alone The pattern matches our human-video findings in one respect and departs from it in another, as gains are larger for the agent than for the objects it manipulates Our method exceeds the skeleton baseline at every budget, reaching 0.546 against 0.527 at 30k hours The gain is, however, roughly constant across budgets: the ceiling is raised, not the rate at which it is approached

Implementation Details

We use AdamW with global batch size 512 and specific learning rates for video-generation and skeleton-projection parameters We evaluate EMA checkpoints on the same 150 held-out clips, without selecting the best checkpoint by validation score Evaluation sequences are captured with a multi-camera stereo platform, and we unproject the target pinhole pixels to match the training set sensor The evaluation protocol involves using 17-frame RGB sequences at 10 fps and 512 × 512 resolution

Data Characteristics

The training set is a large-scale collection of egocentric video of humans performing everyday manipulation activities, recorded from a head-mounted camera The corpus is diverse along every axis we measured: 116 environments, 1,106 scenes, 32,051 tasks, and 14,019 participants Coverage is not concentrated in a few dominant categories; the most common environment category (home) accounts for only 8.

Improvements for AI systems

  1. A new supervision scheme, object-centric adaptive noise scheduling, can be introduced to shift capacity from scene appearance toward object dynamics, which is the only intervention we test that raises the level object fidelity converges to. This allows for measuring object fidelity on its own and discover its saturation point rather than relying solely on overall scene metrics.

  2. A dynamic-region map, Mdyn, can be computed by combining 3D displacement and tracking reliability with a soft spatial prior around projected hands to identify regions where object dynamics are most relevant. This map is then used to raise the noise level or reweight the loss, effectively focusing training on areas where interaction occurs.

  3. The agent's body configuration can be explicitly conditioned using a skeleton sequence, as this design shows that Better conditioning substitutes for most of the agent gains: 1k hours of skeleton-conditioned training exceed the agent fidelity that standard conditioning reaches at 30k. This allows for achieving higher initial agent fidelity more cheaply.

  4. The system can be fine-tuned on ground-truth demonstrations to transfer pre-training, as Ego-centric human pre-training transfers, improving every measure and most clearly hand SCS (+0.07, Table 2). This enables the model to perform better in downstream humanoid settings without needing extensive ego-centric data collection for that specific task.

  5. The model can be evaluated on an out-of-distribution benchmark using structural consistency metrics like SCS, which segments and tracks key scene elements through the predicted and ground-truth videos. This provides a more rigorous assessment of action effects on the world compared to perceptual metrics alone.

Abstract

World models offer a promising alternative to physics-based simulators, yet remain far from practical deployment. We ask how far scaling ego-centric human video takes them, using a dataset of 30,000 hours spanning over 1,000 scene types and 14,000 contributors. Rather than relying on opaque downstream metrics, we directly evaluate agent and object-interaction fidelity on a challenging out-of-distribution benchmark. Increasing training data by 100x improves both, but unevenly: the agent is modeled well, while object fidelity remains far lower and improves slowly. We show that the agent gains need not come from data, and a careful visual conditioning design saturates fidelity with a fraction of it, which lets us measure object fidelity on its own and discover its saturation point. We then introduce a supervision scheme that shifts capacity from scene appearance toward object dynamics, improving object fidelity though a substantial gap remains. Finally, our conclusions transfer to downstream humanoid modeling. Overall, our results suggest that scaling ego-centric data brings agent modeling close to its limit while leaving its effects on the world far behind, and that closing this gap will depend on how models are trained, not only on how much data they see.

Sources

Related papers