KeyRec is a training-free method to compress visual tokens in vision-language models (VLMs) for long videos and streams. It maintains two memory components: a dense recent cache for fine-grained current observations and a structured event bank for historical events, with an online add–merge–evict update to keep memory bounded. When a question comes, a text-only router allocates a fixed readout budget between the two stores.
Across four benchmarks and three VLM backbones, KeyRec uses only 10% of the dense visual-token budget and achieves the best compressed performance in 13 of 15 settings. It outperforms strong baselines by 2.21 to 18.37 points on real-time questions. This is interesting because it addresses a key practical bottleneck in scaling VLMs to long videos without training.