linear attention
Linear attention speeds up processing by using a fixed-size memory state that updates with each token, avoiding the quadratic cost of standard softmax attention. This makes it efficient for long sequences but introduces trade-offs in memory and expressivity.
The papers here target those trade-offs. Some compress vision tokens, quantize memory, or add external storage to improve long-range recall. Others enhance the state's expressiveness with new update rules or tunable access patterns. A common finding is that these innovations let linear attention match or surpass softmax performance while preserving its efficiency.
- Papers
- 25
- Released code
- 0
- First seen
- July 2026
- Latest
- Nov 2026
23 papers in the last two months, against 2 in the two before.
Start here
Three papers to read first, in the order they came out- 1Backward pass for long-context AI models runs up to 1.8 times faster
ReWY reduces delta-rule attention's backward local matrix products from seven to three, speeding complete backward passes 1.15 to 1.81 times on NVIDIA GPUs.
Nov 2026 - 2Language models can train up to 24% faster on Google chips
Systolic Linear Attention runs 1.6x to 2.5x faster on TPUs than existing gated delta attention by widening keys and grouping decay gates.
Nov 2026 - 3Language models train faster than Transformers while improving recall
Muir linear attention uses sensitivity-allocated low-rank gating and matched-filter readout to reduce perplexity and improve long-context recall over Gated DeltaNet-2 and Transformer.
Nov 2026
How fast it is moving
Often studied with
hybrid attentionCombining different attention types (e.g., linear and full) or positional/semantic strategies in one model to balance efficiency, accuracy, and memory.7 togethersparse attentionSelectively attending to only the most relevant positions or tokens to reduce computational overhead.4 togetherkv cache compressionReducing the size of the key-value cache in transformers via selective retention, low-rank updates, or adaptive compression.4 togetherWho is working on it
The papers
Most central to this idea first, not most recent.
- Academiccs.LG
Language models train faster than Transformers while improving recall
Muir linear attention uses sensitivity-allocated low-rank gating and matched-filter readout to reduce perplexity and improve long-context recall over Gated DeltaNet-2 and Transformer.
University of Texas at Austin · Nov 2026
- Academiccs.LG
Backward pass for long-context AI models runs up to 1.8 times faster
ReWY reduces delta-rule attention's backward local matrix products from seven to three, speeding complete backward passes 1.15 to 1.81 times on NVIDIA GPUs.
Tianjin University · Nov 2026
- Big Techcs.LG
Language models can train up to 24% faster on Google chips
Systolic Linear Attention runs 1.6x to 2.5x faster on TPUs than existing gated delta attention by widening keys and grouping decay gates.
Google · Nov 2026
- Big Techcs.CL
Routing by key variance improves multi-state linear attention models
Tracking both mean and variance of key-query alignment improves aggregation in multi-state linear attention, boosting long-context performance.
Princeton University, Google DeepMind · Oct 2026
- Chinese Techcs.CV
Training-free vision token compression method tailored for linear attention architectures
V-CoLA compresses vision tokens in linear-attention VLMs, retaining 99.5% performance with 50% tokens and achieving 1.86-6.15x speedup.
Alibaba Cloud Computing, Alibaba Group · Oct 2026
- Top Universitycs.LG
Switching attention layers close gap between linear and softmax
Switching Linear Attention matches or exceeds softmax expressivity while keeping linear attention's fixed-size recurrent state.
Stanford University · Oct 2026
- Big Techcs.LG
Training-free quantization halves memory traffic for linear attention states
LeapQuant quantizes recurrent states to 8-bit with near-lossless accuracy, reducing memory and speeding inference in linear attention models.
UC Berkeley, University of Washington · Oct 2026
- Big Techcs.CL
Language models can process long prompts faster on Google’s chips
TPU-specific attention kernels that reuse matrix operands, cut vector work and overlap computation make softmax attention up to 4.14 times faster and speed up model serving.
New York University, New York University Shanghai · Nov 2026
- Top Universitycs.AI
Sparse replay memory extends linear attention to million-token contexts
NoteBank, a training-free episodic memory, improves long-context retrieval in linear attention from 1.7% to 70.3% at one million tokens.
King Abdullah University of Science and Technology (KAUST) · Oct 2026
- Independentcs.LG
Targeted memory updates improve stable linear attention over long contexts
The Kronecker Delta Rule updates a matrix memory along one direction, yielding bounded stable memory and less interference for long-context linear attention.
Oct 2026
- Chinese Techcs.LG
Two-sided associative memory correction boosts linear attention performance
Gated Slot Attention-2 corrects both key and value memory in linear attention using complementary update rules.
The Hong Kong University of Science and Technology (Guangzhou), Tencent · Oct 2026
- Independentcs.LG
Random Fourier features make Gaussian process attention linear-time and well calibrated
Random Fourier feature Gaussian process attention provides linear-time complexity and calibrated uncertainty for transformer models.
Oct 2026
- Academiccs.LG
Compressed AI memory can be scored directly, speeding generation 2.5x
GaugeRoPE enables direct latent key-value attention without per-token reconstruction, achieving 5.85× KV compression on Mistral-7B.
National University of Defense Technology · Nov 2026
- Research Labcs.LG
A compact memory model captures behavior across 137 experiments
A small cognitive core (CORE) jointly fitted to 137 experiments accounts for much of human behavior.
Institute for Human-Centered AI, Helmholtz Munich, Max Planck Institute for Biological Cybernetics · Oct 2026
- Top Universitycs.LG
Sparse attention mechanism achieves zero-shot extrapolation to longer sequences
FreeSolo achieves zero-shot length extrapolation with linear-time sparse attention.
National University of Singapore, University of Toronto · Oct 2026
- Top Universitycs.AI
Tunable attention masks enable subquadratic long-context sequence modeling
SMat-Attention uses VC dimension as a tunable parameter to control long-range token routing for efficient sequence modeling.
Georgia Institute of Technology, University of California, Berkeley · Oct 2026
- Research Labcs.AI
Language models struggle with state tracking beyond the simplest cases
No language model architecture robustly handles even intermediate levels of state tracking.
Allen Institute for AI, New York University · Nov 2026
- Big Techcs.LG
Attention-aware KV cache placement extends SSD lifetime without sacrificing cache hit rate
Gandalf increases SSD lifetime by 7.3x for KV cache offloading without reducing cache hit rate.
The University of Chicago, TensorMesh Inc. · Oct 2026
- Chinese Techcs.LG
Shared-prefix training accelerates reinforcement learning for hybrid-attention agents
HARTS accelerates agentic RL training on hybrid-attention models by compressing shared rollout-tree prefixes and planning compact packed executions.
Ant Group · Sept 2026
- Chinese Techcs.AI
Parallel training method speeds language model inference by 3.5x
SpecOPD decouples on-policy supervision from sequential rollouts, enabling parallel distillation for speculative decoding.
Alibaba, University of Science and Technology of China · Nov 2026
- Independentcs.LG
Recurrent-state quantization fails at long decode horizons due to scale-state feedback
Per-token dynamic scaling of recurrent state quantization fails over long decode due to scale-state feedback; persistent scaling outperforms.
Independent Researcher · Oct 2026
- Big Techcs.CV
Hybrid attention enables high-quality video generation on a single GPU
Hybrid linear-softmax attention with attention residuals matches full-softmax video quality at linear-time cost.
NVIDIA · July 2026
- Big Techcs.LG
Replaying cached inputs cuts state memory traffic in SSM decoding
Representing an SSM state as a checkpoint plus a recent-input buffer cuts state memory traffic, enables cheap speculative rollback, and keeps state quantization error out of later updates.
Princeton University, NVIDIA · Oct 2026
- Big Techcs.LG
Adaptive attention pruning compresses language-model caches without sacrificing accuracy
Universal Attention achieves over 10x KV-cache compression while improving downstream performance.
IBM Research, SSAIL Lab · Oct 2026
- Big Techcs.LG
Growing neural networks learn by retrieving stored training examples
Replacing linear attention in neural networks with kernelized attention enables growing networks that store all training data.
Google, MIT CSAIL · Oct 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.