The authors developed Universal Attention, a framework that augments standard softmax attention with trainable decay terms derived from first-order and second-order statistics of stored keys.
Adaptive attention pruning compresses language-model caches without sacrificing accuracy
Universal Attention learns which past tokens to discard while retaining softmax attention and RoPE positional embeddings.
Big Tech
Davis Wertheimer · Haochen Shen · Ahan Gupta · Derrick Liu · Yu Chin Fabian Lim · Mudhakar Srivatsa · +3 more
IBM Research · SSAIL Lab · University of Illinois Urbana-Champaign
Research Digest··2 min read
Wertheimer and colleagues introduce a transformer attention layer whose learned decay mechanism doubles as an input-dependent rule for pruning the key-value cache used during generation.
Why this paper
From IBM Research and 2 others
In one line
Universal Attention achieves over 10x KV-cache compression while improving downstream performance.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§