← All concepts

linear attention

Linear attention speeds up processing by using a fixed-size memory state that updates with each token, avoiding the quadratic cost of standard softmax attention. This makes it efficient for long sequences but introduces trade-offs in memory and expressivity.

The papers here target those trade-offs. Some compress vision tokens, quantize memory, or add external storage to improve long-range recall. Others enhance the state's expressiveness with new update rules or tunable access patterns. A common finding is that these innovations let linear attention match or surpass softmax performance while preserving its efficiency.

Papers
25
Released code
0
First seen
July 2026
Latest
Nov 2026

23 papers in the last two months, against 2 in the two before.

Start here

Three papers to read first, in the order they came out
  1. 1Backward pass for long-context AI models runs up to 1.8 times faster

    ReWY reduces delta-rule attention's backward local matrix products from seven to three, speeding complete backward passes 1.15 to 1.81 times on NVIDIA GPUs.

    Nov 2026
  2. 2Language models can train up to 24% faster on Google chips

    Systolic Linear Attention runs 1.6x to 2.5x faster on TPUs than existing gated delta attention by widening keys and grouping decay gates.

    Nov 2026
  3. 3Language models train faster than Transformers while improving recall

    Muir linear attention uses sensitivity-allocated low-rank gating and matched-filter readout to reduce perplexity and improve long-context recall over Gated DeltaNet-2 and Transformer.

    Nov 2026

Who is working on it

Google 2Princeton University 2New York University 2NVIDIA 2

The papers

Most central to this idea first, not most recent.

Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.