Learned sparse attention first scores earlier tokens with a lightweight indexer, then performs attention over only the top-k candidates, the k highest-scoring entries.
Reusing token selections speeds up training with million-token attention windows
HyScale avoids storing full selection-score tables and repeating token selection, delivering a reported 2.07-fold end-to-end training speedup on 512 GPUs.
Chinese Tech
Yangsheng Deng · Xinjing Huang · Yulong Wang · Yunzhuo Liu · Jun Hu · Shuyong Tang · +5 more
University of Edinburgh · Tencent Hunyuan
Research Digest··2 min read
Deng and colleagues present HyScale, a training system that tackles a bottleneck in learned sparse attention: choosing which earlier tokens each query should attend to.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.