The authors studied selection-based compression of the key-value (KV) cache, the stored attention information a model uses to consult earlier tokens.
Refreshing compressed draft caches preserves alignment during long language model outputs
RIFT combines recent tokens, importance-based context selection and compression-aware training to retain over 98% of full-cache acceptance length in one long-context configuration.
Big Tech
Dor Tsur · Maor Ashkenazi · Maximilian Golub · Peter Hicks · Yaniv Galron · Talor Abramovich · +4 more
NVIDIA
Research Digest··2 min read
Tsur and colleagues study how to shrink the memory used by draft models in speculative decoding, where a larger model verifies a smaller model’s proposed tokens.
§