Long-context language models weigh earlier words as they write. weighs only a chosen few, and a small scoring step, the indexer, decides which. The indexer must finish before attention can start, and on long inputs that wait dominates.
Neighbouring layers usually pick nearly the same words, so each layer can reuse the previous layer's pick and run its own scan alongside attention instead of before it.
Decoding became up to 14.5 percent faster without retraining or new weights, and accuracy stayed within a few points of the previous method.