The authors first analyze the behavior of attention across layers in LLMs, finding that while exact top-κ indices change between layers, most changes occur near the boundary of previously selected regions.
Coarse-to-fine sparse attention boosts long-context LLM inference speed
Prism Attention replaces static cross-layer index sharing with dynamic refinement, achieving 3.4x speedup over FlashAttention at 128K tokens.
Chinese Tech
Zhenxu Tian · Kebin Liu · Zhengwu Yang · Yi Su · Qingqing Dang · Yanlin Sha · +5 more
Soochow University · Baidu Inc.
Research Digest··3 min read
Prism Attention is a sparse attention mechanism for LLM decoding that reduces the overhead of selecting relevant past tokens across layers.
§