Language models can search long contexts without comparing every token

The authors report that a tree-based search preserves evaluated model capabilities through 256,000-token contexts, while a performance model projects better computational scaling at much greater lengths.

Industry

Leon Lufkin · Cutter Dawes · Alexander Speicher · Kamesh Krishnamurthy · Tomás Figliolia · Beren Millidge

Zyphra Research

Research Digest·2 November 2026·3 min read

Illustration: Language models can search long contexts without comparing every token

In one line

Zyphra Hierarchical Attention uses a tree of compressed key summaries to achieve subquadratic scaling for attention in large language models.

Before you read this

Two ideas this paper builds on

sparse attentionSelectively attending to only the most relevant positions or tokens to reduce computational overhead.36 papers →kv cache compressionReducing the size of the key-value cache in transformers via selective retention, low-rank updates, or adaptive compression.52 papers →

Read it as

Attention in large language models compares every word with every other word. This becomes extremely slow and memory-heavy when the input is very long, preventing models from handling very long documents or conversations.

The authors designed a method that organizes key summaries in a tree structure. When a query comes, it travels down the tree to find only the most relevant keys, skipping most of the context. This reduces the total work dramatically.

The method maintains good performance on complex reasoning and retrieval tasks while scaling subquadratically with context length, much better than the usual quadratic growth. This makes it possible to extend useful context windows to millions of tokens without crippling costs.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

sparse attention long-context processing kv cache compression knowledge distillation