Competition Domains Matter: Selective Head Globalization for Training-Free Sparse Attention in Video Diffusion Transformers
cs.CV
Submitted: 2026-10-10
Updated: 2026-10-10
License: http://creativecommons.org/licenses/by/4.0/
The gist: Training-free sparse attention is a practical route to efficient video diffusion transformer inference, yet existing methods typically fix the competition domain at each query row for every attention
Terminology
Abstract
Training-free sparse attention is a practical route to efficient video diffusion transformer inference, yet existing methods typically fix the competition domain at each query row for every attention head. Prior work primarily addresses how to estimate token-level importance within this fixed domain; whether the domain itself should adapt to head heterogeneity remains under-specified. We introduce EG-Sparse, a training-free framework that routes low-entropy heads to a grouped cross-head pool while retaining per-row local selection for high-entropy heads, using only scores already materialized by the sparse pipeline. A controlled diagnostic fixes the ranking score and varies only the competition rule: grouped cross-head selection reduces low-entropy-head reconstruction error by 9.8%, whereas forcing high-entropy heads into an all-head global pool increases it by 7.9%. Across four configurations, EG-Sparse achieves higher video quality while reducing attention density and GPU denoising latency compared with strong training-free sparse-attention baselines. Mechanism ablations further show that selective domain adaptation outperforms uniform, random, and fixed domain-assignment policies. These results support competition-domain adaptation as a distinct design axis within centroid-guided training-free sparse attention.
Sources
- SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs
- HunyuanVideo: A Systematic Framework For Large Video Generative Models
- Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generation
- Radial Attention: $O(n\log n)$ Sparse Attention with Energy Decay for Long Video Generation
- DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidance
- LiteAttention: A Temporal Sparse Attention for Diffusion Transformers
- Wan: Open and Advanced Large-Scale Video Generative Models
- USV: Unified Sparsification for Accelerating Video Diffusion Models
- DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads
- Training-Free Efficient Video Generation via Dynamic Token Carving
- PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models
- HEART: Exploiting Head Heterogeneity in Sparse Attention for Video Diffusion
- Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
- Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models