They developed FlashMask-3, building on FlashMask's column-wise sparse mask representation.
Mask-efficient distributed attention speeds up long-context training by up to 3.7x
FlashMask-3 uses a column-wise sparse mask representation with zero-communication mask localization to accelerate structured attention on H100 GPUs.
Chinese Tech
Guoxia Wang · Qianyue He · Siming Wu · Haoyang Xie · Qiwen Bao · Jinle Zeng · +3 more
Baidu Inc.
Research Digest··2 min read
Wang et al.
Why this paper
From Baidu Inc.
In one line
FlashMask-3 improves throughput for long-context training with structured attention masks on Hopper GPUs.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§