The authors built a rolling-agent workload where documents are evicted and appended while the rest of the prompt remains unchanged, mimicking long-running research or financial agents.
Selective KV-cache reuse depends on request history in rolling agent workloads
Document-aligned recomputation at a 5% budget reduces answer variation across request orders from 69.0% to 26.1% compared to token top-k.
Top University
Tiffany Gu · Annie Guan · Manshu Huang · Nitin Rao · Siddhant Shah · Margaret Capetz
Carnegie Mellon University · Capital One · University of Chicago · University of California, Berkeley · University of Southern California
Research Digest··3 min read
Thread:KV Cache Compression
The authors show that when a KV cache persists across multiple LLM calls in a rolling agent workload, the same prompt can produce different answers depending on the sequence of earlier requests.
Why this paper
From Carnegie Mellon University and 5 others · Part of KV Cache Compression, now 7 papers
In one line
Persistent KV-cache reuse makes outputs depend on request history, while contiguous recomputation substantially improves consistency without sacrificing the reported latency gain.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§