← All concepts

looped transformer

A transformer is a neural network that reads sequences in layers. A looped transformer reuses the same block of layers several times, feeding each pass's output back in as the next input. That adds computation and depth without new parameters; loop count can be chosen per request. The danger is unstable states, so what gets carried around the loop matters.

The papers find that naive repetition is not enough. Learned ways to mix old and new states, compressing or sparsifying the carried state, and using an early loop's prediction as a contrast make later loops genuinely useful. One checkpoint then serves many depths: deeper loops improve accuracy, memory stays bounded, and users trade latency for quality.

Papers
27
Released code
2
First seen
Oct 2026
Latest
Nov 2026

27 papers in the last two months, against 0 in the two before.

Start here

Three papers to read first, in the order they came out
  1. 1Halting heads can overstate compute savings in looped language models

    Halting heads in looped language models trained with mixture loss save almost no compute under their own exit rule.

    Oct 2026
  2. 2Reusing layers in trained AI models pays off only for the smallest models

    A loop added after training pays for its extra inference compute only for the smallest models; larger models gain nothing over equal-parameter or equal-compute baselines.

    Nov 2026
  3. 3Encoder loop adds compute without growing memory in language models

    Looped CEDs achieve recurrent depth with zero round-wise KV growth and match looped Transformer quality.

    Nov 2026

Who is working on it

Apple 2ELLIS Institute Tübingen 2Max Planck Institute for Intelligent Systems 2

The papers

Most central to this idea first, not most recent.

Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.