VHA halves the number of physical query heads while keeping only two KV heads per layer, creating large groups that map to tensor cores.
Attention design cuts KV cache and compute while improving LLM quality.
Virtual Head Attention uses two foldable linear operations to outperform GQA, MLA, and MFA across scales up to a 30B-A3B MoE model.
Chinese Tech
Xibo Ding · Guoxia Wang · Jinle Zeng · Jiabin Yang · Sirui Tao · Chao Yang · +1 more
Baidu Inc. · Peking University
Research Digest··3 min read
The authors introduce Virtual Head Attention (VHA), which combines Q Premix and Linear Postmix to recover attention diversity from halved physical query heads.
Why this paper
From Baidu Inc. and Peking University
In one line
Virtual Head Attention cuts KV cache, training FLOPs, and improves quality via two linear operations folded into GQA-2.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ✓Compute or model size stated (params 30B-A3B MoE)
- ·No stated limitations found
- ✓Reports numbers on named benchmarks (2 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§