The paper shows that during text generation, a language model's internal states already contain signals that predict whether expanding attention to the full context will help predict the next token. The authors exploit this to design On-Demand Attention (ODA), a method that first computes local (windowed) attention and only recomputes with full global attention when a lightweight learned 'recall head' predicts a significant benefit.
ODA is interesting because it requires training only a small recall head, leaving the pretrained model unchanged and the full KV cache intact for later use. Experiments on Qwen and Gemma models show that ODA recovers most of the performance lost under pure local attention while substantially reducing the number of global attention calls, leading to measured speedups in vLLM.