LampAttention is a hardware-algorithm co-design that adapts FlashAttention to mixed precision.
Adaptive mixed-precision attention matches full precision with selective 16-bit recomputation
By adaptively rerouting only numerically sensitive attention sub-blocks to 16-bit precision, the method preserves model performance while enabling a hardware co-design for efficient long-context inference.
Chinese Tech
Stanislav Budzinskiy · Marian Gloser · Tolunay Yilmaz · Ying Hong Tham · Yuanyi Lin · Wenyi Fang · +2 more
University of Vienna · Huawei Heisenberg Research Center · Huawei Technologies Co. Ltd
Research Digest··2 min read
Thread:Precision Scaling Laws
The authors introduce LampAttention, a mixed-precision FlashAttention variant that uses 8-bit arithmetic for most attention computations and selectively recomputes a minority of sub-blocks in 16-bit.
Why this paper
From Huawei Heisenberg Research Center and 2 others · Part of Precision Scaling Laws, now 4 papers
In one line
LampAttention adaptively uses 8-bit precision for most attention sub-blocks and recomputes sensitive ones in 16-bit without performance loss.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§