Tuesday 15 September 2026
Evening EditionRSS

ZOTPAPER

News without the noise

ZOTPAPER
Creative Agents

Hybrid attention enables high-quality video generation on a single GPU

By combining linear and softmax attention with block attention residuals, a 5B parameter model matches full-softmax quality while running 3.2x faster.

PaperBig Techcs.CVarXiv:2607.21553v1 ↗
Junsong Chen · Jincheng Yu · Yitong Li · Shuchen Xue · Haozhe Liu · Jingyu Xin · +8 more

NVIDIA

Research Digest··2 min read
Read the paper →
The authors introduce SANA-Video 2.0, a video diffusion transformer that uses a hybrid of linear and softmax attention to achieve efficient long-sequence processing. The model, scaled to 5B and 14B parameters, generates 720p video on a single H100 GPU and achieves a VBench score of 84.30 at 480p in 13.2 seconds, matching the quality of larger softmax-based models while being significantly faster.

what_they_did

The authors propose a hybrid linear-softmax attention mechanism for video diffusion transformers. They combine gated linear attention (O(N) complexity) with periodic gated-softmax anchors at a 3:1 ratio to restore full-rank token interactions. They also introduce Block Attention Residuals (AttnRes) to propagate refreshed representations across layers, boosting deep-layer effective rank by ~12%. The model is trained from scratch on video data, with proxy studies at reduced resolution to determine the optimal 25% softmax ratio. They evaluate at 5B and 14B scales, using 40-step sampling and a compiled DiT forward pass.

key_findings

  • The hybrid attention achieves quality comparable to full-softmax video DiTs (VBench 84.30 at 480p) while maintaining linear attention's favorable long-sequence scaling.
  • The compiled DiT forward pass is 3.2x faster than a matched full-softmax baseline at 720p/60 frames, and the gap expands with longer videos.
  • With full-stack Sol-Engine optimization (kernel fusion, caching, sparse attention), the 5B pipeline runs in 13.06 seconds at 720p/5 seconds, making it 120x faster than Wan 2.2-A14B on a single H100.
  • The block attention residuals boost effective rank in deep layers by ~12%, improving representational capacity.

why_it_matters

This work demonstrates that hybrid attention can recover the expressiveness of full softmax attention at substantially lower computational cost, enabling scalable long, high-resolution video generation on limited hardware. The approach challenges the assumption that quadratic attention is necessary for high-quality video diffusion, offering a practical path for deployment.

caveats

The evaluation is limited to a single GPU (H100) and specific benchmarks. The proxy studies for the softmax ratio were done at reduced resolution and may not perfectly generalize to full resolution. The comparison to Wan 2.2-A14B is a single speed benchmark; quality comparisons are limited. The model is trained from scratch, so the method may not directly apply to fine-tuning existing pretrained models.

§

Analysis

The paper fits into the broader trend of making diffusion models more efficient for video generation, a computationally intensive task. Previous work has explored linear attention (e.g., Video LDM, etc.) but often at a quality cost. SANA-Video 2.0's hybrid approach is a clever compromise, and the residual connections for attention blocks are a novel contribution. The results are impressive in terms of speedup, though the quality comparison is only against a specific baseline (Wan 2.2) which may not be the state-of-the-art. The paper is well-executed and provides a clear methodology.

newspaper

Research Digest

Articles published under the Zotpaper byline are synthesized from multiple source publications by our AI editor and reviewed by our editorial process. Each story combines reporting from credible outlets to give readers a balanced, comprehensive view.

Editorial ProcessSubscribe