SLA2: Sparse-Linear Attention with Learnable Routing and Quantization-aware Training
cs.LG, cs.AI, cs.CV
Submitted: 2026-02-13
Updated: 2026-10-18
License: http://creativecommons.org/licenses/by/4.0/
The gist: Sparse-Linear Attention (SLA) combines sparse and linear attention to accelerate diffusion models and has shown strong performance in video generation.
Terminology
Abstract
Sparse-Linear Attention (SLA) combines sparse and linear attention to accelerate diffusion models and has shown strong performance in video generation. However, (i) SLA relies on a heuristic split that assigns computations to the sparse or linear branch based on attention-weight magnitude, which can be suboptimal. Additionally, (ii) after formally analyzing the attention error in SLA, we identify a mismatch between SLA and a direct decomposition into sparse and linear attention. We propose SLA2, which introduces (I) a learnable router that dynamically selects whether each attention computation should use sparse or linear attention, (II) a more faithful and direct sparse-linear attention formulation that uses a learnable ratio to combine the sparse and linear attention branches, and (III) a sparse + low-bit attention design, where low-bit attention is introduced via quantization-aware fine-tuning to reduce quantization error. Experiments show that on video diffusion models, SLA2 can achieve 97% attention sparsity and deliver an 18.6x attention speedup while preserving generation quality.
Sources
- Mixture of Contexts for Long Video Generation
- Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- Separate, Dynamic and Differentiable (SMART) Pruner for Block/Output Channel Pruning on Computer Vision Tasks
- LongNet: Scaling Transformers to 1,000,000,000 Tokens
- SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs
- Identifying Sensitive Weights via Post-quantization Integral
- Residual Context Diffusion Language Models
- Cascadia: An Efficient Cascade Serving System for Large Language Models
- FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference
- Radial Attention: $O(n\log n)$ Sparse Attention with Energy Decay for Long Video Generation
- Twilight: Adaptive Attention Sparsity with Hierarchical Top-$p$ Pruning
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusion
- MoBA: Mixture of Block Attention for Long-Context LLMs
- Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models
- SparQ Attention: Bandwidth-Efficient LLM Inference
- VORTA: Efficient Video Diffusion via Routing Sparse Attention
- Retentive Network: A Successor to Transformer for Large Language Models
- DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Training
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks