The authors developed KV-Kaizen, a learned selector that configures an LLM's key-value cache before the prompt-processing, or pre-fill, stage.
Adaptive per-layer choices shrink LLM caches without sacrificing accuracy
KV-Kaizen learns context-dependent combinations of cache sharing, reduced precision and rank truncation under a specified memory budget.
Big Tech
Joao Monteiro · Louis Béthune · Anastasiia Filippova · Sonia Laguna · David Grangier · Marco Cuturi
Apple
Research Digest··2 min read
Monteiro et al.
Why this paper
From Apple
In one line
KV-Kaizen learns a per-layer, context-adaptive compression selector along precision, rank, and depth axes, preserving accuracy while reducing cache size by up to 32x.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§