Speculative Forcing for Efficient Autoregressive Video Generation

The gist The introduction states that Speculative Forcing, a framework that predicts when each token’s estimate has sufficiently converged, allows individual tokens to be accepted at different steps while continuing to refine only where needed

How it works

Speculative Forcing is introduced as a framework that treats intermediate clean estimates as drafts and predicts when individual tokens can be safely accepted early Once accepted, a token is frozen as its final output and undergoes no further denoising, while its cached keys and values remain available as context for the remaining tokens This requires predicting whether a token’s current draft is already close to its full-step output without running the remaining denoising steps

The framework addresses this challenge with two verifiers

1. Training-free verifier: This verifier uses similarity to previously generated content and stability between consecutive estimates of the same token to provide evidence of convergence without additional training It scores each token by the distance from its draft to the most similar clean latent token at the first step, or by how much two consecutive drafts differ from each other on subsequent steps

2. Learned verifier: This verifier leverages contextual cues encoded in the pretrained generator’s intermediate features to predict each token’s remaining gap to its full-step output A shallow head trained with the generator frozen predicts each token’s gap to the full-step output, showing that the intermediate features contain cues about how much its predictions will change with further denoising

Token-wise Commitment

The process involves several steps for commitment and continued denoising

Commitment and continued denoising:

Every token starts active, and the first step produces a draft for every token At each step k < K, the verifier evaluates the active tokens An accepted token i has its output fixed to xˆ(k)0(i) and is excluded from all subsequent denoising forwards Its layerwise keys and values computed at commitment remain cached, allowing active tokens to continue attending to it Rejected tokens are re-noised to the next noise level and continue denoising Any tokens still active at step K are committed at that step The completed chunk combines estimates accepted at potentially different steps, ct(i) = ˆx(k∗(i))0(i)

Convergence Criterion

To decide which drafts can be accepted, the convergence criterion is measured by the convergence gap relk(i) = (xˆ(K)0(i) - xˆ(k)0(i)) / σt, for k < K A draft meets the convergence criterion when relk(i) ≤ ϵ This gap serves as an oracle and is used as the ground truth in eq. (2)

Evaluation

Speculative Forcing was evaluated on LongLive (Yang et al., 2025a) and the world model LingbotWorld 2.0 (Gao et al., 2026) On both short and long videos, our method maintains the quality of the four-step generator with approximately 1.7 denoising steps per token, running over 1.5× faster at 41.7 and 37.8 fps On controllable world model rollouts, a harder setting where camera motion continually changes the visible content, it maintains quality with approximately 2.5 denoising steps per token and runs 1.35× faster at 24.5 fps

Conclusion

Speculative Forcing allows individual tokens to stop denoising at different steps By treating intermediate clean estimates as drafts, it predicts which tokens can be accepted early and continues denoising only the remaining tokens Training-free and learned verifiers enable these decisions without modifying the generator’s weights Experiments on short and long videos and world model rollouts demonstrate faster generation while maintaining quality, with comparable or lower quality variation and preserved camera-following accuracy in world modeling

Limitations

The achievable acceleration depends on both the underlying generator and the difficulty of the generation domain If the backbone is sensitive to reducing denoising steps, fewer tokens can be accepted early without degrading generation quality, which limits the attainable speedup Similarly, more challenging domains may require additional refinement across a larger fraction of tokens

Potential Applications

Token-wise commitment may be particularly effective when much of the target content is already constrained by available conditions In talking-head generation (Huang et al., 2026), for example, the background can remain largely unchanged while motion is concentrated around the face and upper body In environment-grounded world models such as Seoul World Model (Seo et al., 2026) and World Observer (Choi et al., 2026), warped scene observations and reference images provide visual evidence for parts of the target view, while uncovered regions require new content to be generated Dexterous World Models (Kim et al., 2025) provide static scene renderings along the camera trajectory and generate interaction-induced changes while preserving unaffected content

Other Discussions

The clean forward pass used to refresh the KV cache adds one full pass per chunk Assuming computation scales linearly with token evaluations, the ideal speed-up relative to the four-step generator is 5/(¯k + 1) In practice, throughput does not scale proportionally with the mean denoising depth ¯k Each chunk continues until its last active token is accepted, incurring forward-pass overhead at every executed step even when few tokens remain active This overhead therefore depends on maxi k∗(i) within each chunk rather than the average across tokens

Additional Results and Analyses

The learned verifier achieves generation quality comparable to oracle verification without computing the full-step reference at inference Spatiotemporal smoothing encourages coherent acceptance decisions across neighboring tokens and achieves lower Drift and higher imaging quality, supporting more stable rollouts The learned verifier predicts the convergence gap itself, setting a tolerance on the predicted discrepancy to the full-step output without retraining or fine-tuning The training-free verifier uses redundancy at the first step and stability after, at the cost of two thresholds The head leads every latent signal throughout

Additional Ablation on Our Head

Training on commitment rollouts or replacing the convolutional head with a per-token MLP yields similar generation quality and throughput Spatiotemporal smoothing encourages coherent acceptance decisions across neighboring tokens and achieves lower Drift and higher imaging quality, supporting more stable rollouts

Verification Accuracy

The learned verifier variants are calibrated to our operating point (ϵ = 0.

1. textbf Treatment of Intermediate Estimates as Drafts (Speculative Forcing): This framework treat[s] intermediate clean estimates as drafts and predict when individual tokens can be safely accepted early. This allows tokens that already closely match their eventual full-step outputs in the initial steps to be frozen as its final output and undergo no further denoising, substantially reducing computation.

2. textbf Training-Free Verification for Low Latency: The training-free verifier uses similarity to clean content at the first step and stability between consecutive estimates of the same token provide evidence of convergence without additional training. This enables acceptance decisions without modifying the generator’s weights while leveraging existing latent information.

3. textbf Learned Feature-Based Gap Prediction: A learned verifier leverages contextual cues encoded in the pretrained generator’s intermediate features to estimate each token’s remaining gap to its full-step output. This allows for prediction of convergence gaps, as shown by the loss function L(ϕ) = (1/N(K − 1)) Σ i=1 N (X−1k=1 K X−1 k=smoothl1 rˆk(i), relk(i), (2).

4. textbf Adaptive Verification Tolerance: The system allows for dynamic control of the speed-quality trade-off by adjusting the tolerance parameter epsilon, as shown in Table 4 where Increasing ε reduces mean denoising steps, with ε = 0.20 and ε = 0.24 maintaining the four-step generator’s quality while achieving lower Drift.

5. textbf Spatiotemporal Smoothing for Coherence: Applying spatiotemporal smoothing operator S to each verifier’s scores before thresholding encourages coherent decisions across neighboring tokens, leading to better results in world models by preventing premature chunk-wise acceptance [from] propagating errors to later chunks.

Few-step autoregressive video generators enable streaming generation, yet they run every token through the final denoising step, regardless of how much refinement it needs. We find this uniform computation is often unnecessary: many tokens already closely match their eventual full-step outputs in the initial steps, while others require refinement until the end. We introduce Speculative Forcing, a framework that treats intermediate clean estimates as drafts and predicts when individual tokens can be safely accepted early. Once accepted, a token is frozen as its final output and undergoes no further denoising, while its cached keys and values remain available as context for the remaining tokens. We explore two alternative verifier designs: a training-free one based on similarity to clean content and stability across steps, and a learned one that reads the frozen generator's intermediate features to estimate each token's remaining gap to its full-step output. Across short-video, long-video, and world generation, Speculative Forcing preserves the quality of full-step sampling while substantially reducing computation. On LongLive, it matches the four-step generator with ~1.7 denoising steps per token, running over 1.5x faster at 41.7 and 37.8 fps. On Lingbot-World 2.0, where camera motion continually changes the visible content, it maintains quality with ~2.5 denoising steps per token and runs 1.35x faster at 24.5 fps.