The authors first studied the properties of KV cache across layers.
Shared latent across layers cuts KV cache memory by 10x with no training
BellowsKV compresses keys and values up to 14x while keeping 97-99% of original accuracy on Llama-3.1-8B and other models.
Academic
Jiamu Zhang · Hanjie Chen
Rice University
Research Digest··2 min read
Jiamu Zhang and Hanjie Chen propose BellowsKV, a training-free method that maps the hidden states of multiple adjacent layers into a single shared latent per token.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.