← All concepts

hybrid attention

The core idea is to mix two different types of attention within a single model. Full attention lets every token look at every other token, which is powerful but gets very slow with long texts. Linear attention or recurrent layers are faster and use less memory, but can be less accurate. Hybrid attention strategically places a few precise full-attention layers among many efficient ones, aiming to get the best of both worlds: strong performance at a lower cost.

The papers show this trade-off is powerful but nuanced. They confirm hybrids can match full-attention accuracy while being much faster, especially for very long contexts. However, switching to a hybrid changes *how* a model finds information, moving from relying on a token's position to its meaning, which can help or hurt depending on the task. New designs are also tackling the limitation of simpler layers, giving them better access to older information without slowing them down, and even using signals from the recurrent layers to smartly decide what to keep in memory.

Papers
20
Released code
2
First seen
Sept 2026
Latest
Nov 2026

19 papers in the last two months, against 1 in the two before.

Start here

Three papers to read first, in the order they came out
  1. 1Approximate scores recover context lost by sparse long-context attention

    PQ-HSA preserves accuracy at low retrieval budgets by reusing approximate ranking scores to include unselected KV-cache tokens while accelerating long-context decoding.

    Oct 2026
  2. 2AI coding model beats predecessor using roughly one-fifth the active parameters

    Nemotron 3.5 Super trains a 12B-active multimodal model that exceeds a 4.6x larger model on SWE-bench Pro.

    Oct 2026
  3. 3On-device AI drafters can accept more tokens with less memory by separating context from local attention

    HyFlash achieves higher acceptance than full-context attention with less KV storage, especially for shallow drafters and long contexts.

    Nov 2026

Who is working on it

NVIDIA 2Ant Group 2Cornell University 2

The papers

Most central to this idea first, not most recent.

Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.