VSA2 is a trainable mechanism for video that reduces computation while maintaining quality. It introduces a fine-grained router that more precisely identifies important tokens, and allows each query to attend to a variable number of key-value pairs. The authors also propose a Hard-to-Easy Curriculum where models trained with high sparsity and evaluated with lower sparsity perform better. On 720p videos, VSA2 accelerates attention by 8.9x and end-to-end generation by 4.62x compared to FlashAttention-3, with comparable or better quality.
Fine-grained routing makes sparse video attention faster without quality loss
VSA2 adaptively allocates attention across video tokens and can replace full attention during ongoing diffusion-transformer pretraining.
Chinese Tech
Peiyuan Zhang · Guoqiang Wei · Yilong Zhao · Zixiang Zhang · Wei Zhou · Will Lin · +4 more
University of California, San Diego · ByteDance Seed · University of California, Berkeley · Georgia Institute of Technology
Research Digest··2 min read
Thread:Sparse Video Attention
In one line
VSA2 makes video DiT attention 8.9x faster with comparable quality by using a fine-grained router and adaptive sparsity.
Before you read this
Three ideas this paper builds onsparse attentionSelectively attending to only the most relevant positions or tokens to reduce computational overhead.51 papers →diffusion transformerA neural network architecture combining transformer layers with diffusion processes for high-quality image and video generation.75 papers →reinforcement learningA machine learning method where an agent learns to make sequences of decisions by receiving rewards or penalties for its actions.296 papers →
Read it as
Zhang et al. introduce VSA2, a trainable sparse-attention system for video diffusion transformers that selects a small, query-dependent subset of token interactions.
VSA2 uses a fine-grained router to estimate which key-value tokens matter for each query.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.