Joint token-feature KV cache compression speeds long-context LLMs
Zihan Teng, Jiayu Zhao, Wentao Ren, Minhao Fan, Tianrui Ma, Song Chen, and Weichen Liu propose SlimKV, a KV-cache compression method that jointly reduces the number of cached states and their per-token dimensions. Using low-rank-aware training, SlimKV compresses long contexts into compact beacon memory states with layer-adaptive rank allocation, and exploits an asymmetric dependence on key-side RoPE to remove its application from beacon keys during decoding. This achieves up to a 7.34x attention speedup and 3.38x end-to-end decoding speedup over the uncompressed model at 128K context length.
