Video Large Language Models (Video LLMs) struggle with inference efficiency due to the massive number of visual tokens from long videos. Recent compression methods use complex token selection, pruning, or merging strategies. This paper investigates how much performance can be obtained simply by preserving the structure encoded in the visual representations. SimpleCluster is a training-free baseline that performs position-aware clustering of visual tokens across frames, then represents each cluster by the mean of its original features. No additional training or architectural changes are needed. Across four video understanding benchmarks (MVBench, EgoSchema, LongVideoBench, VideoMME) and three Video LLMs, SimpleCluster matches or outperforms recent compression methods at various token retention ratios, with strong robustness even at 1% retention. Analysis shows that preserving the global coverage of the original visual feature distribution is a key factor for downstream performance.
Simple clustering preserves video understanding with far fewer visual tokens
Across four benchmarks and three video language models, a training-free clustering method matched or outperformed more elaborate compression approaches, including at 1 percent token retention.
Chinese Tech
Xiao Zhang · Wang Zeng · Sheng Jin · Wentao Liu · Chen Qian · Shichao Kan
Central South University · SenseTime Research · Tetras.AI
Research Digest··3 min read
Thread:Token Pruning Efficiency
In one line
Simple clustering of visual tokens by position-aware cross-frame features preserves structure and matches or beats complex compression methods.
Before you read this
Two ideas this paper builds oncontext compactionCompressing or summarizing an LLM's past context into a smaller representation to extend memory and improve reasoning within limited token budgets.81 papers →vision-language modelA model that handles tasks combining visual and textual information, such as editing images from conversation or reasoning across images and text.265 papers →
Read it as
Zhang et al. introduce SimpleCluster, a baseline that compresses video representations by grouping similar visual tokens across frames and replacing each group with its mean feature.
SimpleCluster operates on visual tokens after they pass through the model's projector, which maps vision-encoder outputs into the language model's representation space.
How it works
stage by stageThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.
What it released
Code