MOSAIC searches all transformer layers jointly using Bayesian optimization guided by a surrogate MLP to find per-layer KV-cache budgets for a given eviction method.
Per-layer KV-cache budgets learned via multi-objective search improve LLM inference efficiency
MOSAIC uses Bayesian optimization to find layer-specific allocations that transfer across cache budgets, outperforming uniform allocation in 146 of 154 settings.
Big Tech
Sravanth Kodavanti · Manjunath Arveti
Samsung Research · Georgia Institute of Technology
Research Digest··2 min read
The authors introduce MOSAIC, a multi-objective search method that learns per-layer key-value cache budgets for LLM inference.
§