← All threads

Subquadratic Attention

Designing attention mask families and architectures that achieve subquadratic complexity for long-context modeling while preserving expressivity.

15 papers

Where this stands

The written synthesis of this thread is for subscribers. Subscribe.

Results across this thread

11 reported results from the papers in this thread.

PaperBenchmarkMetricResultConditionsCode

The table is for subscribers. Subscribe to see every reported number side by side.

How this thread developed

  1. Oct 2026 · The Hong Kong University of Science and Technology (Guangzhou), Tencent

    Two-sided associative memory correction boosts linear attention performance

    Presents Gated Slot Attention-2 (GSA2) which adds two-sided associative memory correction to linear attention, outperforming subquadratic baselines at 1.3B scale.

  2. 4 further papers

    Oct 2026 · Stanford University

    Switching attention layers close gap between linear and softmax

    Blends linear and softmax attention states within a single layer to achieve the memory efficiency of subquadratic mechanisms with expressivity closer to full softmax attention.

  3. Oct 2026 · Amazon AGI

    Token graph communities speed long-context decoding without model retraining

    Introduces a sparse attention mask family that achieves subquadratic complexity via token graph community pruning for long-context generation.

  4. Oct 2026 · Independent Researcher, University of Illinois at Urbana-Champaign

    Spectral theory diagnoses two failure modes in RoPE long-context models

    identifies two failure modes in RoPE for long-context models using spectral theory, informing attention design.

  5. 1 further paper

    Oct 2026

    Temporal indexer prediction speeds up long-context sparse attention by 14.9%

    SPIN sparsifies the indexer in sparse attention by predicting block importance from past scores, achieving block sparsity and improving throughput.

  6. 4 further papers

    Oct 2026 · Independent Researcher

    Recurrent-state quantization fails at long decode horizons due to scale-state feedback

    Exposes a previously unknown feedback loop between quantization and scale-state dynamics that degrades long-context performance in linear attention and SSMs.

6 of 15 papers shown