Long-context generation repeatedly accesses a key-value cache, the stored representations of earlier tokens.
Summarizing unread attention tokens preserves accuracy while reading one percent
Yu and colleagues report that tail proxies retain 99% of dense-attention accuracy on a long-context benchmark while accelerating the decode attention stage.
Big Tech
Hyunwoo Yu · Jongbeom Lee · Hyunbin Kim · Sungjun Park · Jaemin Choi · Jincheol Yang · +5 more
Sogang University · Samsung Electronics · Pusan National University
Research Digest··2 min read
Yu and colleagues introduce OpenAttention, a training-free method that reads selected entries from a language model’s key-value cache and summarizes the rest with representative tokens.
§