MLA stores a compressed representation of each token’s keys and values, called a latent, alongside a smaller positional key.
Role-aware quantization compresses latent attention caches with little accuracy loss
Across four MLA models, the authors report full-cache storage averaging about 2.5 bits per element while keeping reasoning and coding scores within 0.5 points of BF16.
Chinese Tech
Tianqi Chen · Xunchao Li · Linchuan Xie · Guanghua Yu · Jianchen Zhu · Peisong Wang · +1 more
University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Tencent
Research Digest··2 min read
Chen et al. introduce RAQ-MLA, a method that compresses the two components of a Multi-Head Latent Attention cache according to their different effects on model output.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.