Subquadratic Attention
Designing attention mask families and architectures that achieve subquadratic complexity for long-context modeling while preserving expressivity.
15 papers
Where this stands
The written synthesis of this thread is for subscribers. Subscribe.
Results across this thread
11 reported results from the papers in this thread.
The table is for subscribers. Subscribe to see every reported number side by side.
How this thread developed
Oct 2026 · The Hong Kong University of Science and Technology (Guangzhou), Tencent
Two-sided associative memory correction boosts linear attention performance
Presents Gated Slot Attention-2 (GSA2) which adds two-sided associative memory correction to linear attention, outperforming subquadratic baselines at 1.3B scale.
4 further papers
Oct 2026 · Stanford University
Switching attention layers close gap between linear and softmax
Blends linear and softmax attention states within a single layer to achieve the memory efficiency of subquadratic mechanisms with expressivity closer to full softmax attention.
Oct 2026 · Amazon AGI
Token graph communities speed long-context decoding without model retraining
Introduces a sparse attention mask family that achieves subquadratic complexity via token graph community pruning for long-context generation.
Oct 2026 · Independent Researcher, University of Illinois at Urbana-Champaign
Spectral theory diagnoses two failure modes in RoPE long-context models
identifies two failure modes in RoPE for long-context models using spectral theory, informing attention design.
1 further paper
Oct 2026
Temporal indexer prediction speeds up long-context sparse attention by 14.9%
SPIN sparsifies the indexer in sparse attention by predicting block importance from past scores, achieving block sparsity and improving throughput.
4 further papers
Oct 2026 · Independent Researcher
Recurrent-state quantization fails at long decode horizons due to scale-state feedback
Exposes a previously unknown feedback loop between quantization and scale-state dynamics that degrades long-context performance in linear attention and SSMs.
6 of 15 papers shown