The authors analyze production-style LLM workloads including multi-round conversations and agentic workflows, and find that KV caches from different attention layer types have different reuse patterns: full-attention states are reused across many rounds, while linear- and sparse-attention states tend to be used once and then discarded.
Attention-aware KV cache placement extends SSD lifetime without sacrificing cache hit rate
Gandalf preferentially admits full-attention KV caches to flash storage over linear- and sparse-attention states, increasing endurance 7.3x in trace-based evaluation.
Big Tech
Maharani A. P. Irawan · Kuntai Du · Jiayi Yao · Dongjoo Seo · Yuyang Huang · Haryadi S. Gunawi · +1 more
The University of Chicago · TensorMesh Inc. · Samsung Semiconductor
Research Digest··2 min read
The authors propose Gandalf, a KV cache offloading system for LLM serving that uses attention layer type as a placement signal when deciding which caches to write to SSD.
§