hybrid attention
The core idea is to mix two different types of attention within a single model. Full attention lets every token look at every other token, which is powerful but gets very slow with long texts. Linear attention or recurrent layers are faster and use less memory, but can be less accurate. Hybrid attention strategically places a few precise full-attention layers among many efficient ones, aiming to get the best of both worlds: strong performance at a lower cost.
The papers show this trade-off is powerful but nuanced. They confirm hybrids can match full-attention accuracy while being much faster, especially for very long contexts. However, switching to a hybrid changes *how* a model finds information, moving from relying on a token's position to its meaning, which can help or hurt depending on the task. New designs are also tackling the limitation of simpler layers, giving them better access to older information without slowing them down, and even using signals from the recurrent layers to smartly decide what to keep in memory.
- Papers
- 20
- Released code
- 2
- First seen
- Sept 2026
- Latest
- Nov 2026
19 papers in the last two months, against 1 in the two before.
Start here
Three papers to read first, in the order they came out- 1Approximate scores recover context lost by sparse long-context attention
PQ-HSA preserves accuracy at low retrieval budgets by reusing approximate ranking scores to include unselected KV-cache tokens while accelerating long-context decoding.
Oct 2026 - 2AI coding model beats predecessor using roughly one-fifth the active parameters
Nemotron 3.5 Super trains a 12B-active multimodal model that exceeds a 4.6x larger model on SWE-bench Pro.
Oct 2026 - 3On-device AI drafters can accept more tokens with less memory by separating context from local attention
HyFlash achieves higher acceptance than full-context attention with less KV storage, especially for shallow drafters and long contexts.
Nov 2026
How fast it is moving
Often studied with
linear attentionAn attention mechanism that computes attention with linear, rather than quadratic, complexity in sequence length.7 togethersparse attentionSelectively attending to only the most relevant positions or tokens to reduce computational overhead.5 togethermixture of expertsA neural architecture with many specialized sub-networks, only a few activated per input via a router, balancing capacity and efficiency.3 togetherWho is working on it
The papers
Most central to this idea first, not most recent.
- Big Techcs.CL
AI coding model beats predecessor using roughly one-fifth the active parameters
Nemotron 3.5 Super trains a 12B-active multimodal model that exceeds a 4.6x larger model on SWE-bench Pro.
NVIDIA · Oct 2026
- Chinese Techcs.LGcode
Approximate scores recover context lost by sparse long-context attention
PQ-HSA preserves accuracy at low retrieval budgets by reusing approximate ranking scores to include unselected KV-cache tokens while accelerating long-context decoding.
The Hong Kong University of Science and Technology, Huawei Technologies Ltd. · Oct 2026
- Academiccs.CL
Hybrid backbones can efficiently adapt into diffusion language models
Hybrid-attention diffusion models from Qwen3.5 adapt faster and decode in any order and in parallel.
University of Texas at Austin · Sept 2026
- Chinese Techcs.LG
Shared-prefix training accelerates reinforcement learning for hybrid-attention agents
HARTS accelerates agentic RL training on hybrid-attention models by compressing shared rollout-tree prefixes and planning compact packed executions.
Ant Group · Sept 2026
- Big Techcs.CL
On-device AI drafters can accept more tokens with less memory by separating context from local attention
HyFlash achieves higher acceptance than full-context attention with less KV storage, especially for shallow drafters and long contexts.
Microsoft Applied Sciences Group · Nov 2026
- Big Techcs.LG
Multi-hop retrieval from a shared KV cache gives hybrid models strictly deeper composition power
Multi-hop retrieval from a shared KV cache lets a single hybrid block compute m-function composition, creating a strict expressivity hierarchy.
Technion – Israel Institute of Technology, NVIDIA · Oct 2026
- Academiccs.CL
Positional encoding choice shifts retrieval from positional to semantic mechanisms
Hybrid positional encodings shift language models from positional to semantic in-context retrieval, trading off performance.
Cornell University · Oct 2026
- Chinese Techcs.DC
AI agents can finish batches in up to 65% less time
Coordinated inference and environment optimizations reduce batch makespan for LLM agent workloads by 10.4% to 65.0%.
Ling Team, Ant Group · Nov 2026
- Big Techcs.LG
Language models can train up to 24% faster on Google chips
Systolic Linear Attention runs 1.6x to 2.5x faster on TPUs than existing gated delta attention by widening keys and grouping decay gates.
Google · Nov 2026
- Top Universitycs.AIcode
Recurrent memory signals predict which KV cache entries hybrid LLMs will need
GateSight uses GDN recurrent memory signals to predict attention KV utility, enabling 90% cache reduction with top accuracy.
Shanghai Jiao Tong University · Oct 2026
- Big Techcs.LG
Attention-aware KV cache placement extends SSD lifetime without sacrificing cache hit rate
Gandalf increases SSD lifetime by 7.3x for KV cache offloading without reducing cache hit rate.
The University of Chicago, TensorMesh Inc. · Oct 2026
- Chinese Techcs.CV
Training-free vision token compression method tailored for linear attention architectures
V-CoLA compresses vision tokens in linear-attention VLMs, retaining 99.5% performance with 50% tokens and achieving 1.86-6.15x speedup.
Alibaba Cloud Computing, Alibaba Group · Oct 2026
- Top Universitycs.LG
Sparse attention mechanism achieves zero-shot extrapolation to longer sequences
FreeSolo achieves zero-shot length extrapolation with linear-time sparse attention.
National University of Singapore, University of Toronto · Oct 2026
- Chinese Techcs.CV
A method that generates consistent video observations across multiple vehicles in a shared driving scene
CoDrive jointly generates world-consistent multi-vehicle driving videos with precise camera trajectory control.
Tsinghua University, Tencent Hunyuan · Oct 2026
- Chinese Techcs.CV
Sparse flow matching generates 3D shapes at ultra-high voxel resolution
Filigree3D generates detailed 3D geometry from one image at 2048^3 voxel resolution by combining sparse computation, local-global attention, and multi-scale visual conditioning.
Alibaba Group · Oct 2026
- Big Techcs.LG
Transformer text generation may never converge, but temperature ensures it
Within a broad fixed-depth transformer class, sufficiently high model-specific softmax temperature guarantees long-run cache convergence, while lower temperatures can permit persistent nonconvergence.
Google Research · Oct 2026
- Industrycs.IR
Shared embeddings let smaller query models search larger models’ document indexes
A 0.6B and a 9B model distilled from one 18B teacher achieve state-of-the-art late-interaction retrieval and can be used asymmetrically.
Perplexity AI · Oct 2026
- Big Techcs.LG
Shared memory improves looped Transformers while shrinking context caches
Shared memory across recursions improves quality and reduces memory in looped transformers.
IBM Research, Cornell University · Oct 2026
- Big Techcs.LG
Training-free quantization halves memory traffic for linear attention states
LeapQuant quantizes recurrent states to 8-bit with near-lossless accuracy, reducing memory and speeding inference in linear attention models.
UC Berkeley, University of Washington · Oct 2026
- Academiccs.LG
Backward pass for long-context AI models runs up to 1.8 times faster
ReWY reduces delta-rule attention's backward local matrix products from seven to three, speeding complete backward passes 1.15 to 1.81 times on NVIDIA GPUs.
Tianjin University · Nov 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.