Programmable World Model
cs.CV
Submitted: 2026-09-09
Updated: 2026-10-13
Comments: Homepage: https://alaya-lab.github.io/pwm GitHub: https://github.com/AlayaLab/pwm
Code: https://github.com/AlayaLab/pwm
Project page: https://alaya-lab.github.io/pwm
License: http://creativecommons.org/licenses/by/4.0/
The gist: Recent video world models generate increasingly realistic and interactive visual experiences, yet lack reliable mechanisms for maintaining persistent world state and enforcing programmable rules over
Terminology
Abstract
Recent video world models generate increasingly realistic and interactive visual experiences, yet lack reliable mechanisms for maintaining persistent world state and enforcing programmable rules over extended interactions. We introduce Programmable World Model, a framework that decouples world-state evolution from visual observation generation. An agent translates natural-language instructions into executable programs that specify entity states and state-transition rules, enabling direct control over individual entities and their interactions. A lightweight engine executes these programs to update and maintain an explicit, persistent global world state, including off-screen entities and non-visual attributes. To connect world state with visual generation, we introduce state-augmented 3D oriented bounding boxes (OBBs) as an intermediate representation. This representation, together with the target camera trajectory, is deterministically compiled into pixel-aligned spatiotemporal conditioning signals for a pretrained video model serving as the generative renderer. This design allows users to create playable games with predefined mechanics, direct control over individual entities, and persistent world state throughout gameplay. We further introduce CombatStateBench, a benchmark for evaluating programmable world models. On CombatStateBench, our method achieves 94% Count Accuracy and 98% State Accuracy, substantially outperforming existing interactive video world models while supporting coherent long-horizon generation. These results demonstrate the effectiveness of separating explicit state evolution from generative rendering for building persistent, programmable worlds.
Sources
- Qwen3-VL Technical Report
- Infinite Worlds with Versatile Interactions
- Coarse-to-Real: Generative Rendering for Populated Dynamic Scenes
- LTX-Video: Realtime Video Latent Diffusion
- ViPE: Video Pose Engine for 3D Geometric Perception
- Generative World Renderer
- Yume: An Interactive World Generation Model
- WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling
- AlayaWorld: Interactive Long-Horizon World Modeling -- Full Technical Report
- Advancing Open-source World Models
- Wan: Open and Advanced Large-Scale Video Generative Models
- VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models