Velox: Spatial Memory Makes World Models Efficient

arXiv:2610.22311 · cs.CV · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Velox: Spatial Memory Makes World Models Efficient".

Jane: The gist The introduction states that context management is a central methodological challenge for video world models,

Tom: First, who's behind it and why it matters.

Paper summary: Jane: So, looking at "Velox: Spatial Memory Makes World Models Efficient", it seems like the authors have successfully tackled the growing cost of context management in chunked video diffusion models by introducing a training-free attention router based on frustum slab correspondences. The main contribution is this soft geometric fusion rule that uses OBB frustum slab overlap to modulate layer importance, while keeping an unbiased fallback for uncovered regions.

Lu: What this suggests for the field is that we can decouple the context selection mechanism from model training, which simplifies deployment significantly. It moves away from frame-level selection toward a more spatial memory approach where relevance is determined by geometric overlap rather than just temporal proximity alone.

Meng: I think for production, the fact that it achieves a four point three six times attention speedup and two point four three times denoising speedup on chunk forty-eight is compelling evidence that this isn't just theoretical overhead reduction; it actually helps with throughput when generating longer scenes like navigation splits on WBench <ref:2610.22311#pg1>.

Tom: Exactly, because we're seeing these tangible performance gains without any heavy training overhead for the router itself. The authors show that their fixed token-block granularity lines up well with tiled execution styles, which is great for efficient GPU utilization.

Lalam: It shows that spatial memory, when mapped correctly to geometry like frustum slabs, can replace the dense KV cache bottleneck effectively in these long rollouts. This paper gives us a concrete way to handle the growing visual context problem without linearly increasing computational cost with rollout length.

Jane: And looking at the title again, "Spatial Memory Makes World Models Efficient," it really hammers home that the efficiency comes directly from how they manage that spatial memory, using geometry as a guide instead of just relying on raw temporal history. It points toward a future where context management is handled by scene structure itself rather than just sequential storage.

Conclusion: Tom: So, we've been looking at Velox, this paper by those authors who developed these video world models that rely on chunked diffusion. The main idea is using spatial memory to handle the context problem better than just a long list of past frames.

Jane: Right, so it’s about how these models keep track of what happened earlier when they generate video sequences, and Velox uses something called frustum slabs to manage that. What does that actually mean for us as listeners?

Lu: It means instead of looking at every single previous frame sequentially which gets really slow, Velox represents those past chunks spatially, like slices of a camera's view—frustums—and it uses the overlap between those slices to decide which past information is most important for the current part we are generating.

Meng: So, it’s a way to selectively remember context based on where the camera is looking and how much that memory overlaps with what you need now. But what about making this actually run fast when we’re shipping models?

Tom: The results show that this approach works without needing any extra training for the router itself. They see a speedup of four point three six times in attention at chunk forty-eight and denoising speeds up by two point four three times. That’s a real number we can talk about.

Jane: That sounds like a big deal because those numbers show that this isn't just theoretical math; it actually makes the generation process much faster for longer tasks, like navigation splits on WBench. So, what does this mean for the future of these AI models?

Lalam: I’m pretty sure what Velox shows is that we can handle long-term visual context by using geometry to guide memory access instead of just relying on a massive key-value cache that grows way too big. It makes the model more efficient in terms of computation per step.

Tom: It suggests a direction for how we think about memory in these models, moving from pure temporal history to spatial awareness within the world representation. We’ll look at how this specific geometry mapping affects other types of visual context selection next.

Qiuhong Shen, Shizun Wang, *Xinchao Wang

National University of Singapore

cs.CV

Submitted: 2026-10-15

Updated: 2026-10-15

Code: https://github.com/mit-han-lab/Block-Sparse-Attention

Project page: https://velox-world.com/velox

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: The gist The introduction states that context management is a central methodological challenge for video world models, and Velox introduces a training-free attention router using frustum slab

Key concepts

Frustum Slab Correspondences: Historical video blocks are treated as 3D camera frustum slabs with defined depth intervals. This geometric representation allows the system to map where past visual information is located in space and time. This spatial mapping is crucial for determining which past blocks are relevant for the current region being generated, enabling targeted context retrieval.
Recency-Priority Multiwarp: This mechanism determines the depth of the current block by prioritizing more recent visual data. It ensures that when calculating the depth of a new chunk, newer information is weighted more heavily than older information. This helps in accurately assessing how much 'new' context is available versus 'historical' context for decision-making.
Soft Modulation: For areas well-covered by historical data, Velox uses frustum slab overlaps to gently adjust the importance of each layer's model output. This soft modulation intelligently blends the influence of past blocks based on their spatial relationship, ensuring a smooth transition between historical context and new generation without abrupt changes.
Chunked Video Diffusion: This is the standard way video world models generate long sequences by processing them in smaller segments or chunks. The challenge is that each chunk needs to remember what came before. Velox optimizes this process by replacing traditional, growing KV caches with a geometric routing system, making context management efficient for long simulations.

Terminology

World Model — 1× in this paper · explained in 7 episodes

Summary

The gist The introduction states that context management is a central methodological challenge for video world models, and Velox introduces a training-free attention router using frustum slab correspondences to reduce growing KV attention cost in chunked video world models.

How it works

Velox addresses the bottleneck of dense KV caches by representing historical blocks as camera frustum slabs with depth intervals and obtaining current-block depth through recency-priority multiwarp The core mechanism involves using these frustum slab correspondences to guide which historical KV token blocks each current region should see For well-covered regions, the overlap softly modulates each layer’s model-derived block importance Uncovered regions retain the model-derived route without geometric bias.

The process for generating a route involves several steps that are implemented asynchronously to speed up performance.

**- Frustum slab construction runs asynchronously with denoising on the same H200. 5. The reconstruction and frustum slab preparation path is on the same order as one DiT denoise step in our setting. 8. The worker is the sole writer to the geometry objects. 9. The main stream waits only on a completion event when a future state is needed. 5. The implementation uses a two-chunk freshness lag. 8. Target chunk c consumes a Velox state built from frustum slabs through chunk c − 2, while the worker prepares states for later chunks. 9. The state remains self-consistent: each worker task first inserts exactly one decoded chunk into the frustum slab table and multiwarp cache, then scores that table against the target chunk’s latent cameras. 5. The worker is the sole writer to the geometry objects. 8. The main stream waits only on a completion event when a future state is needed. 9. The state remains self-consistent: each worker task first inserts exactly one decoded chunk into the frustum slab table and multiwarp cache, then scores that table against the target chunk’s latent cameras. 5. The worker is the sole writer to the geometry objects. 8. The main stream waits only on a completion event when a future state is needed. 9. The state remains self-consistent: each worker task first inserts exactly one decoded chunk into the frustum slab table and multiwarp cache, then scores that table against the target chunk’s latent cameras. 5. The worker is the sole writer to the geometry objects. 8. The main stream waits only on a completion event when a future state is needed. 9. The state remains self-consistent: each worker task first inserts exactly one decoded chunk into the frustum slab table and multiwarp cache, then scores that table against the target chunk’s latent cameras. 5. The worker is the sole writer to the geometry objects. 8. The main stream waits only on a completion event when a future state is needed. 9. The state remains self-consistent: each worker task first inserts exactly one decoded chunk into the frustum slab table and multiwarp cache, then scores that table against the target chunk’s latent cameras. 5. The worker is the sole writer to the geometry objects. 8. The main stream waits only on a completion event when a future state is needed. 9. The state remains self-consistent: each worker task first inserts exactly one decoded chunk into the frustum slab table and multiwarp cache, then scores that table against the target chunk’s latent cameras. 5. The worker is the sole writer to the geometry objects. 8. The main stream waits only on a completion event when a future state is needed. 9. The state remains self-consistent: each worker task first inserts exactly one decoded chunk into the frustum slab table and multiwarp cache, then scores that table against the target chunk’s latent cameras. 5. The worker is the sole writer to the geometry objects. 8. The main stream waits only on a completion event when a future state is needed. 9. The state remains self-consistent: each worker task first inserts exactly one decoded chunk into the frustum slab table and multiwarp cache, then scores that table against the target chunk’s latent cameras. 5. The worker is the sole writer to the geometry objects. 8. The main stream waits only on a completion event when a future state is needed. 9. The state remains self-consistent: each worker task first inserts exactly one decoded chunk into the frustum slab table and multiwarp cache, then scores that table against the target chunk’s latent cameras. 5. The worker is the sole writer to the geometry objects. 8. The main stream waits only on a completion event when a future state is needed. 9. The state remains self-consistent: each worker task first inserts exactly one decoded chunk into the frustum slab table and multiwarp cache, then scores that table against the target chunk’s latent cameras. 5. The worker is the sole writer to the geometry objects. 8. The main stream waits only on a completion event when a future state is needed. 9. The state remains self-consistent: each worker task first inserts exactly one decoded chunk into the frustum slab table and multiwarp cache, then scores that table against the target chunk’s latent cameras. 5. The worker is the sole writer to the geometry objects. 8. The main stream waits only on a completion event when a future state is needed. 9. The state remains self-consistent: each worker task first inserts exactly one decoded chunk into the frustum slab table and multiwarp cache, then scores that table against the target chunk’s latent cameras. 5. The worker is the sole writer to the geometry objects. 8. The main stream waits only on a completion event when a future state is needed. 9. The state remains self-consistent: each worker task first inserts exactly one decoded chunk into the frustum slab table and multiwarp cache, then scores that table against the target chunk’s latent cameras. 5. The worker is the sole writer to the geometry objects. 8. The main stream waits only on a completion event when a future state is needed. 9. The state remains self-consistent: each worker task first inserts exactly one decoded chunk into the frustum slab table and multiwarp cache, then scores that table against the target chunk’s latent cameras. 5. The worker is the sole writer to the geometry objects. 8. The main stream waits only on a completion event when a future state is needed. 9.

Improvements for AI systems

The improved system can achieve significant speedups in long-horizon video generation by replacing dense attention with Velox's routing mechanism, reaching a 4.36× speedup in attention and a 2.43× denoising speedup at chunk 48.
The system will maintain visual consistency over long rollouts by using region-level context selection, as Velox replaces historical blocks as camera frustum slabs with depth intervals and obtains current-block depth through recency-priority multiwarp.
The improved system can preserve local motion continuity without relying on complex cross-chunk bookkeeping, utilizing the safeguard: Previouschunk retention and global-aligned Gilbert neighborhoods preserve local motion continuity without a separate cross-chunk seam table.
The system will adapt to different scene dynamics by using a flexible routing strategy: In well-covered regions, frustum slab overlap softly modulates each layer’s model-derived block importance, while for uncovered regions, it retain the model-derived route without geometric bias.
The system can improve spatial and photometric consistency over dense attention in complex scenes by utilizing soft fusion: For a well-covered query block with ρi ≥ ηcov, we first suppress overlap values at or below the noise threshold τ and rescale the remaining values, thereby promoting correspondence without imposing a hard replacement set.

Abstract

Video world models turn actions and camera motion into long visual rollouts, making video a natural interface for interactive simulation. Such interaction requires responsive generation, yet coherent rollouts must retain visual memory of what has already been generated. Chunked video diffusion models usually keep this context in growing KV caches, so later chunks become progressively slower. To address this bottleneck, we introduce Velox, a training-free attention router for region-level context selection using frustum slab correspondences from streaming geometry. Velox represents historical blocks as camera frustum slabs with depth intervals and obtains current-block depth through recency-priority multiwarp. For well-covered regions, frustum slab overlap softly modulates each layer's model-derived block importance. Uncovered regions retain the model-derived route without geometric bias. The fixed token-block granularity matches FlashAttention-style tiled execution and reduces the attention term that grows with rollout length. Frustum slab construction runs asynchronously with denoising on the same H200. At chunk 48, Velox accelerates attention by 4.36 times and denoising by 2.43 times. On the full 158-case WBench navigation split, it remains within 0.15 API-free average points of dense LingBot-Fast-Cam. Please see our project page at https://velox-world.com/velox/ for more details.