sparse attention
Attention normally lets each word look at every previous word, costing huge compute as text grows. Sparse attention instead picks only a few important tokens per word, cutting that cost. But deciding which tokens matter can itself be slow and can leak privacy, because the pattern of what gets skipped reveals information about the content. The key idea is to keep the benefits, speed, long-context handling, while avoiding the pitfalls: selecting important tokens cheaply, preserving quality, and protecting data.
Across the papers, the common thread is that sparse attention is no longer just a speed hack; it is being reworked as a principled, trained or training-free method that reduces computation without hurting output. Several approaches focus on better selection, using learned routers, community detection, or predictive indexers, while others target specific domains like video, where attention patterns repeat. All report large speedups and note trade-offs in accuracy or privacy, showing the technique is mature enough for real systems but still needs care.
- Papers
- 51
- Released code
- 4
- First seen
- Sept 2026
- Latest
- Nov 2026
50 papers in the last two months, against 1 in the two before.
Start here
Three papers to read first, in the order they came out- 1Million-token prompts served at 2.3 times the throughput without retraining
Retrofitting pretrained dense-attention large language models with chunk-level sparse attention, without retraining, speeds long-context serving while keeping benchmark scores close to dense attention.
Nov 2026 - 2Overlapping indexer scan with attention speeds up long-context decoding by up to 14%
A sparse-attention layer can attend the token selection its previous layer's indexer already made, letting the scan overlap attention and speeding up long-context decoding.
Nov 2026 - 3Sparse attention for long-context AI verifies its own quality in one pass
Coverage-Verified Sparse Attention achieves closed-loop quality control by using the softmax mass fraction to verify and correct sparse attention outputs.
Nov 2026
How fast it is moving
Often studied with
kv cache compressionReducing the size of the key-value cache in transformers via selective retention, low-rank updates, or adaptive compression.20 togetherdiffusion transformerA neural network architecture combining transformer layers with diffusion processes for high-quality image and video generation.12 togetherflow matchingA generative training method that learns an ordinary differential equation mapping noise to data by matching velocity fields along probability paths.6 togetherWho is working on it
The papers
Most central to this idea first, not most recent.
- Academiccs.LG
Sparse attention for long-context AI verifies its own quality in one pass
Coverage-Verified Sparse Attention achieves closed-loop quality control by using the softmax mass fraction to verify and correct sparse attention outputs.
National Taiwan University · Nov 2026
- Big Techcs.LG
Million-token prompts served at 2.3 times the throughput without retraining
Retrofitting pretrained dense-attention large language models with chunk-level sparse attention, without retraining, speeds long-context serving while keeping benchmark scores close to dense attention.
NVIDIA · Nov 2026
- Industrycs.LG
Overlapping indexer scan with attention speeds up long-context decoding by up to 14%
A sparse-attention layer can attend the token selection its previous layer's indexer already made, letting the scan overlap attention and speeding up long-context decoding.
Writer AI Research · Nov 2026
- Independentcs.CL
Sparse attention trained on 32K tokens works at million-token contexts
RecSA achieves million-token length extrapolation with logarithmic compute cost using a tree-structured KV cache.
Nov 2026
- Research Labcs.CV
Video AI models generate clips with 99% fewer attention computations
ESTRA accelerates video diffusion models up to 5.39× by computing only 1% of attention exactly.
The University of Sydney, CSIRO · Nov 2026
- Independentcs.AI
Clustering keys makes 10-million-token attention 22x faster with similar accuracy
Nov 2026
- Industrycs.CL
Language models can search long contexts without comparing every token
Zyphra Hierarchical Attention uses a tree of compressed key summaries to achieve subquadratic scaling for attention in large language models.
Zyphra Research · Nov 2026
- Independentcs.AI
Language models process million-token prompts up to 1.8 times faster
Routing each query through a causal inverted-file index over indexer keys recovers more of the exhaustive top-k than block-based selection at higher prefill throughput.
Nov 2026
- Top Universitycs.CV
Video generators can run twice as fast without retraining
Accurate Block Attention (AB-Attention) skips 89-92% of attention in video diffusion transformers while improving PSNR by up to 4.4 dB.
InfRec, The Chinese University of Hong Kong, Shenzhen · Oct 2026
- Big Techcs.LG
Speech language models can drift as audio retrieval mistakes accumulate
C3 outperforms other KV retrieval methods on long-form speech tasks by clustering audio keys and having heads compete.
University at Buffalo, Amazon AGI · Nov 2026
- Research Labcs.CL
Precomputing attention indices from earlier layers speeds long-context AI without accuracy loss
EarlySelectKV precomputes key-value cache selections using earlier hidden states, reducing long-context inference latency without degrading accuracy.
Institute of Semiconductors, Chinese Academy of Sciences, University of Chinese Academy of Sciences · Nov 2026
- Big Techcs.CL
Language models using diffusion can run 2.78 times faster
Diffusion language models run faster when the sparse attention mask is refreshed periodically during denoising and each attention head gets its own budget.
University of Washington, University of Waterloo · Nov 2026
- Chinese Techcs.AI
Reusing token selections speeds up training with million-token attention windows
HyScale speeds up long-context training for learned sparse attention by 2.07x on 1M-token sequences.
University of Edinburgh, Tencent Hunyuan · Oct 2026
- Big Techcs.CL
Skipping selected attention gradients speeds long-context fine-tuning while preserving dense inference
LazyFlash speeds up long-context fine-tuning by making the backward pass block-sparse while keeping the forward pass dense.
Microsoft · Oct 2026
- Independentcs.CL
Document parser reads hundreds of pages without growing per-step compute
Unlimited OCR 2 achieves constant computational cost across arbitrarily long documents, enabling one-shot hundred-page parsing.
Oct 2026
- Big Techcs.LG
Language models can retain near-full accuracy reading 1% of cached memory
A single count-aware mean proxy for the attention tail yields lower error than sampling and achieves 99% of dense accuracy at a 1% read rate.
Sogang University, Samsung Electronics · Oct 2026
- Chinese Techcs.CV
Image generators create less distorted panoramas with fewer visible seams
CubeAtlas representation enables distortion-free and seamless 360-degree panorama generation with better quality and lower attention computation than equirectangular projection.
Tencent · Oct 2026
- Chinese Techcs.LG
Mask-efficient distributed attention speeds up long-context training by up to 3.7x
FlashMask-3 improves throughput for long-context training with structured attention masks on Hopper GPUs.
Baidu Inc. · Oct 2026
- Independentcs.LG
Temporal indexer prediction speeds up long-context sparse attention by 14.9%
SPIN predicts which KV cache blocks are important using past scores, reducing indexer overhead by 30-40% and improving throughput by up to 14.9%.
Oct 2026
- Academiccs.LG
Sparse attention in LLMs leaks private data via GPU side channels
Sparse attention leaks prompt attributes and generated tokens through GPU cache and TLB contention, enabling attacks with average success rates of 90.9% and 87.3%.
Shandong University, Xi'an Jiaotong University · Oct 2026
- Chinese Techcs.CL
Coarse-to-fine sparse attention boosts long-context LLM inference speed
Prism Attention achieves 3.4× decoding speedup over FlashAttention by replacing rigid index reuse with a proposal-refined sparse attention framework.
Soochow University, Baidu Inc. · Oct 2026
- Big Techcs.LG
Token graph communities speed long-context decoding without model retraining
CommunityKV uses graph partitioning on attention scores to achieve up to 1.71x throughput for long-context decoding.
Amazon AGI · Oct 2026
- Top Universitycs.LG
Sparse attention mechanism achieves zero-shot extrapolation to longer sequences
FreeSolo achieves zero-shot length extrapolation with linear-time sparse attention.
National University of Singapore, University of Toronto · Oct 2026
- Chinese Techcs.LG
Sharing key-value data across queries speeds sparse-attention prefill
Neighboring queries in sparse-attention prefill share KV entries; jointly processing them and reusing shared entries reduces kernel latency by up to 55.1%.
Huawei Technologies Co., Ltd. · Oct 2026
- Chinese Techcs.LGcode
Approximate scores recover context lost by sparse long-context attention
PQ-HSA preserves accuracy at low retrieval budgets by reusing approximate ranking scores to include unselected KV-cache tokens while accelerating long-context decoding.
The Hong Kong University of Science and Technology, Huawei Technologies Ltd. · Oct 2026
- Chinese Techcs.CV
Cached token selection narrows diffusion transformers’ sparse attention quality gap
MC-Sparse achieves up to 2.32× denoising speedup over dense attention in diffusion transformers with negligible quality loss.
Fudan University, Tencent HY · Oct 2026
- Chinese Techcs.CV
Periodic stripe structures in video DiT attention enable efficient sparse computation
Parameterized stripe attention exploits recurring video DiT attention patterns to accelerate HunyuanVideo and Wan 2.1 by 1.57× and 1.37× with limited quality loss.
Alibaba Group · Oct 2026
- Chinese Techcs.CV
Interaction-aware sparse attention and timestep-adaptive sparsity accelerate video generation
iCATS achieves state-of-the-art efficiency-quality trade-off in video generation through interaction-aware sparse attention and SNR-guided sparsity.
Hangzhou Normal University, T-Head Semiconductor Co., Ltd. · Oct 2026
- Top Universitycs.CL
Routing attention heads by token enables joint scaling of parameters and context
NAMOH activates K of H attention heads per token, so scaling heads shortens routed contexts and reduces KV access without increasing storage.
Peking University · Oct 2026
- Chinese Techcs.CV
Fine-grained routing makes sparse video attention faster without quality loss
VSA2 makes video DiT attention 8.9x faster with comparable quality by using a fine-grained router and adaptive sparsity.
University of California, San Diego, ByteDance Seed · Oct 2026
- Chinese Techcs.CV
CoVisco compresses long videos inside the vision encoder
CoVisco uses abstract tokens and segmented attention to compress visual tokens, enabling 400-token video understanding that matches or exceeds OneVision-Encoder.
ERNIE Team, Baidu Inc., The Hong Kong University of Science and Technology · Oct 2026
- Top Universitycs.CV
High-resolution image editing gets a 4x speedup by keeping a low-resolution anchor
FLASH enables faithful 2K/4K image editing by anchoring a high-resolution detail branch to a frozen 1K editing backbone.
Shanghai Innovation Institute, Fudan University · Nov 2026
- Chinese Techcs.LG
Fitting regression models to tokens improves compression in two settings
Test-time ridge regression compresses token sequences, outperforming pooling and projection on retrieval and visual merging.
The Chinese University of Hong Kong, Independent Researcher · Oct 2026
- Research Labcs.LGcode
Training-free inference framework trims redundant computation and memory in looped Transformers
FlashLoop achieves up to 1.64x speedup and 6x KV-cache reduction in looped transformers without accuracy loss.
ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems · Oct 2026
- Chinese Techcs.CV
Sparse flow matching generates 3D shapes at ultra-high voxel resolution
Filigree3D generates detailed 3D geometry from one image at 2048^3 voxel resolution by combining sparse computation, local-global attention, and multi-scale visual conditioning.
Alibaba Group · Oct 2026
- Industrycs.LG
A Vestigial Attention Branch Signals Which Cache Entries to Keep
On NoPE-MLA models, a 64-dimensional decoupled branch carries a query-independent salience signal that keeps needle retrieval at 1.00 under 8x and 0.96 under 32x compression.
Yotta Labs · Sept 2026
- Top Universitycs.CV
Speculative decoding speeds up vision models by 3.58 times with visual-token weighting
MMFlash speeds up vision-language model inference by up to 3.58x using adaptive weight adjustments for visually dependent tokens.
National University of Singapore · Nov 2026
- Top Universitycs.LG
AI weather model matches rival's accuracy using one-tenth the training compute
Astra matches GenCast's quarter-degree forecast skill using 9.8% of its training compute and 39 times fewer inference FLOPs.
Seoul National University · Oct 2026
- Big Techcs.CL
Refreshing compressed draft caches preserves alignment during long language model outputs
RIFT compresses draft-model KV caches by mixing prefill and recent tokens, refreshing selection, and training with sparse masks to preserve acceptance length.
NVIDIA · Oct 2026
- Big Techcs.LG
Multi-hop retrieval from a shared KV cache gives hybrid models strictly deeper composition power
Multi-hop retrieval from a shared KV cache lets a single hybrid block compute m-function composition, creating a strict expressivity hierarchy.
Technion – Israel Institute of Technology, NVIDIA · Oct 2026
- Chinese Techcs.CV
One-step diffusion framework restores 1080p video in real time with efficient streaming attention
FastVR restores high-resolution video at 11 FPS using a one-step diffusion model with a lightweight VAE and chunk-wise causal attention.
Alibaba Group, Xidian University · Oct 2026
- Academiccs.LGcode
AI forecasting model adapts to sensor changes, guarantees uncertainty at new locations
OmniST forecasts at unseen sensor locations with guaranteed uncertainty coverage via topology-invariant design and conformal calibration.
Indian Institute of Technology Patna, Sopra Banking Software India · Nov 2026
- Big Techcs.CV
AI models’ attention reveals only part of how they select images
Image retrieval in vision-language models concentrates in sparse attention heads, and the heads that matter most for multi-image answers both attend to and write the target image's content.
Seoul National University, Upstage AI · Nov 2026
- Chinese Techcs.AI
Parallel training method speeds language model inference by 3.5x
SpecOPD decouples on-policy supervision from sequential rollouts, enabling parallel distillation for speculative decoding.
Alibaba, University of Science and Technology of China · Nov 2026
- Big Techcs.CV
Video generators can nearly halve history storage while improving reconstruction quality
A compression scheme for cached attention state in autoregressive video models preserves generation quality better by tuning Keys to match downstream outputs rather than to reconstruct the raw cache.
Zhejiang University, NVIDIA · Nov 2026
- Top Universitycs.AIcode
Recurrent memory signals predict which KV cache entries hybrid LLMs will need
GateSight uses GDN recurrent memory signals to predict attention KV utility, enabling 90% cache reduction with top accuracy.
Shanghai Jiao Tong University · Oct 2026
- Chinese Techcs.CV
Stochastic attention routing enables efficient RL alignment for diffusion models
Routing Policy Optimization uses temporary stochastic attention routes to improve reward-aligned visual generation with less training compute while preserving dense deterministic inference.
National University of Singapore, ByteDance · Oct 2026
- Chinese Techcs.CV
StreamErase enables real-time streaming video object removal at 26 FPS
StreamErase enables sub-second, real-time streaming video object removal at over 26 frames per second on a single GPU.
Wuhan University, Alibaba Group · Oct 2026
- Big Techcs.LG
Adaptive attention pruning compresses language-model caches without sacrificing accuracy
Universal Attention achieves over 10x KV-cache compression while improving downstream performance.
IBM Research, SSAIL Lab · Oct 2026
- Top Universitycs.AI
Tunable attention masks enable subquadratic long-context sequence modeling
SMat-Attention uses VC dimension as a tunable parameter to control long-range token routing for efficient sequence modeling.
Georgia Institute of Technology, University of California, Berkeley · Oct 2026
- Top Universitycs.CV
Predicting velocity fields helps video models generate more realistic fluids
FluidGen generates physically plausible fluid videos by jointly predicting a 3D velocity field alongside RGB.
Stanford University · Oct 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.