Language models train faster than Transformers while improving recall

The authors’ fixed-memory models improve prediction and retrieval, train 2.7 times faster than a Transformer at 32,000-token sequences, and preserve near-perfect passkey retrieval to one million tokens.

Academic

Tushaar Gangavarapu · Gil Tabak · Clemens JS Schaefer

University of Texas at Austin

Research Digest·6 November 2026·2 min read

Illustration: Language models train faster than Transformers while improving recall

In one line

Muir linear attention uses sensitivity-allocated low-rank gating and matched-filter readout to reduce perplexity and improve long-context recall over Gated DeltaNet-2 and Transformer.

Before you read this

Three ideas this paper builds on

linear attentionAn attention mechanism that computes attention with linear, rather than quadratic, complexity in sequence length.24 papers →context compactionCompressing or summarizing an LLM's past context into a smaller representation to extend memory and improve reasoning within limited token budgets.75 papers →kv cache compressionReducing the size of the key-value cache in transformers via selective retention, low-rank updates, or adaptive compression.67 papers →

Read it as

Ordinary attention grows more expensive as text grows. compresses the past into a fixed-size memory, but many facts in the same memory overwrite each other. A newer design erases old answers before writing new ones, yet treating erasure and writing as equally important wastes capacity.

Muir analyzes where capacity actually goes: writing matters far more than erasing. It spends fewer parameters on erasure, widens the feed-forward network that stores associations, re-weights old memory rows so they do not dominate, and keeps training stable.

At the largest size tested, Muir lowers perplexity, a prediction-quality measure, by about 2 percent over a strong baseline, and keeps near-perfect retrieval after a million tokens, making long-context models cheaper and more reliable.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

linear attention sensitivity-allocated gating matched-filter readout gated delta-rule models context compaction kv cache compression