The authors developed ReadKV, which encodes every cached key and value progressively, so longer bit prefixes produce increasingly precise reconstructions.
Query-adaptive cache reads preserve accuracy while moving fewer bits
ReadKV retains eight-bit key-value cache entries but fetches only the precision each decoding query is predicted to need.
Big Tech
Siddharth Bhandari · Lucas Gretta · Krishna Balasubramanian · Shiva Kasiviswanathan
Amazon · University of California, Berkeley
Research Digest··3 min read
Bhandari et al.
Why this paper
From Amazon and University of California, Berkeley
In one line
ReadKV uses progressive KV cache codes and query-dependent bit allocation to reduce memory reads while maintaining perplexity within 0.66% of uncompressed.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ✓Reports numbers on named benchmarks (2 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§