5B parameters on FineWeb-Edu.
Kronecker product factorization compresses key-value caches up to 10x without accuracy loss.
KPA matches or exceeds MHA, MQA, GQA, MLA, and TPA across four model scales and nine benchmarks under identical training conditions.
Chinese Tech
Bahareh Nikpour · Hossein Rajabzadeh · Maryam Dialameh · Vahid Partovi Nia · Parsa Omidi
Huawei Technologies · University of Waterloo · Huawei Noah's Ark Lab
Research Digest··2 min read
The authors propose Kronecker Product Attention (KPA), a structured attention mechanism that caches keys and values as compact Kronecker product factors.
Why this paper
From Huawei Technologies and 2 others
In one line
Kronecker Product Attention compresses KV caches by representing keys and values as Kronecker products of small factors, cutting memory up to 10x while matching or beating prior attention variants.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§