The authors argue that existing evaluations of KV cache reuse, which aim to reduce latency in retrieval-augmented generation by reusing precomputed chunks, are misleading. The standard practice of reporting aggregate accuracy scores over entire datasets includes queries that the model can answer without context, or that the full-prefill baseline cannot answer correctly, inflating the apparent effectiveness of reuse techniques. They show that up to 42% of a method's reported F1 score on a common benchmark comes from queries the baseline cannot answer at all.
They also identify dataset structural problems: chunks in test queries are often semantically clustered, making it too easy to precompute cross-attention, and there is limited cross-query chunk reuse, preventing evaluation of warm reuse methods. To address this, they propose a new evaluation methodology that focuses on a 'meaningful subset' of queries and introduce Boxoffice, a tool that generates synthetic datasets with controlled reuse patterns. They demonstrate that a reuse method's score can swing from 0.98 to 0.00 depending solely on the context in which a chunk was originally cached.