CRONOS: Benchmarking Counterfactual Physical Consistency in Video Models

Video prediction is increasingly viewed as a path toward generalizable world models, yet it remains unclear whether these systems learn underlying causal structure or merely exploit superficial visual correlations for future prediction. We introduce CRONOS, an intervention-based benchmark to evaluate counterfactual physical consistency: whether a model's predictions of physical events respond appropriately to controlled changes in the visual input, such as variations of camera viewpoint, scene, object category, and object appearance. Built in a photorealistic Unreal Engine environment, CRONOS enables controlled, high-fidelity generation of videos across diverse scenes and dynamics, including sampling multiple physically plausible outcomes of the same physical event. In contrast to previous benchmarks, that evaluate video prediction on scenes CRONOS systematically intervenes on four factors -camera viewpoint, scene, object category, and object appearance- while keeping the underlying physical event type, such as a fall, collision, or occlusion, fixed. Our evaluation of recent open-source video generators reveals substantial failures in counterfactual physical consistency: prediction quality for the same physical event type is affected by appearance, environment, and particularly viewpoint changes. CRONOS provides a controlled and reproducible testbed for diagnosing how generated video quality changes across interventions, establishing a concrete target for developing models that perform consistently across changes of multiple conditions. The dataset and evaluation code are available at our project page.