Hybrid attention enables high-quality video generation on a single GPU

By combining linear and softmax attention with block attention residuals, a 5B parameter model matches full-softmax quality while running 3.2x faster.

Big Tech

Junsong Chen · Jincheng Yu · Yitong Li · Shuchen Xue · Haozhe Liu · Jingyu Xin · +8 more

NVIDIA

Research Digest·23 July 2026·2 min read

In one line

Hybrid linear-softmax attention with attention residuals matches full-softmax video quality at linear-time cost.

Before you read this

Two ideas this paper builds on

attention residualResidual connections in attention mechanisms retain past states for efficient, continuous improvement in transformer models.3 papers →linear attentionAn attention mechanism that computes attention with linear, rather than quadratic, complexity in sequence length.18 papers →

Read it as

SANA-Video 2.0 introduces a hybrid video diffusion transformer that combines mostly with periodic softmax anchors at a 3:1 ratio. It also uses Block to route completed block summaries across layers, boosting deep-layer effective rank by ~12%.

At 5B and 14B scales, this model matches the quality of full-softmax video DiTs while being significantly faster. On a single H100, it achieves a VBench score of 84.30 in 13.2s at 480p, and its compiled forward pass is 3.2x faster than a full-softmax baseline at 720p/60s. With additional system-level optimizations, the pipeline runs 120x faster than Wan 2.2-A14B on one GPU.

This is interesting because it makes high-quality long, high-resolution video generation feasible on a single GPU, overcoming the quadratic cost barrier of pure softmax attention.

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

hybrid linear-softmax attention attention residual efficient video generation video diffusion transformer linear attention