The authors trained a lightweight recall head — a small MLP — to predict, from the current decoding state, whether a full global attention pass over the KV cache would significantly improve the next-token prediction.
Selective attention calls, guided by model's own state, speed long-context inference
On-Demand Attention uses a lightweight recall head to predict when global attention is needed, reducing computation without retraining the base model.
Top University
Haibo Feng · Ruiqi Liang · Hanyang Peng · Shiqi Yu
Southern University of Science and Technology · Peking University · Peng Cheng Laboratory
Research Digest··2 min read
The authors introduce On-Demand Attention (ODA), a decoding method that uses a trained recall head to decide when to invoke global attention based on the model's decoding state.
Why this paper
From Peking University and 2 others · Part of Context Engineering for Agents, now 23 papers
In one line
On-Demand Attention selectively invokes global attention during decoding based on predicted benefit, reducing compute with minimal quality loss.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§