Sparse attention trained on 32K tokens works at million-token contexts

Recursive sparse attention with local position encoding outperforms all baselines on RULER from 131K to 2M and accelerates prefill by 3.2x at 524K.

Independent

Ruitian Wang · Xinyu Yang · Minshen Zhang · Hao Zhang

Research Digest·6 November 2026·3 min read

Illustration: Sparse attention trained on 32K tokens works at million-token contexts

In one line

RecSA achieves million-token length extrapolation with logarithmic compute cost using a tree-structured KV cache.

Before you read this

Two ideas this paper builds on

sparse attentionSelectively attending to only the most relevant positions or tokens to reduce computational overhead.48 papers →kv cache compressionReducing the size of the key-value cache in transformers via selective retention, low-rank updates, or adaptive compression.67 papers →

Read it as

Transformer 'attention' grows slower with longer text. Smarter methods currently scan every past chunk first.

RecSA organizes past text into a tree of summaries. A query selects the best summary at the top, then drills down, finding the exact words in a number of steps that grows remarkably slowly with text length. A mechanism keeps the relative distances seen during training the same, no matter how long the context gets.

On a standard long-context test, RecSA scored 60.5, far ahead of the best existing method which scored 40.0. It is faster than standard attention from contexts far shorter than a million up to the longest tested, making analysis of entire books in one prompt practical.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

sparse attention long-context processing knowledge distillation kv cache compression context window