Large language models store past keys and values in memory to generate text, but this memory grows with each new word, eventually slowing down and using too much GPU space. Compression reduces memory but often requires rebuilding the original keys before computing attention, adding more work per step.
GaugeRoPE is a training-free method that compresses keys and values into a compact form that can be used directly for attention, without reconstructing the original. It uses a design that respects how the model encodes position, enabling efficient scoring from the compressed state.
On Mistral-7B, GaugeRoPE compresses the cache by 5.85 times while keeping output quality close to the original, and speeds up text generation by up to 2.48 times. This makes it practical for deploying large models with limited memory.
