The authors identify a representation-utilization gap: when visual tokens are aggressively pruned, greedy decoding fails but repeated sampling from the same sparse context recovers correct answers, showing that useful visual evidence remains but is not reliably used. They propose SCOPD, a self-distillation framework where a model with pruned tokens (student) generates reasoning trajectories while a model with full visual context (teacher) supervises those same prefixes. SCOPD+ selectively distills only the response positions most sensitive to visual evidence. At 10% visual-token retention, the vanilla model retains 86.37% of its unpruned performance across 13 benchmarks; SCOPD raises this to 90.49% and SCOPD+ to 92.43%.
Self-distillation helps vision-language models reason with far fewer visual tokens
SCOPD trains models on pruned visual inputs using full-context teacher guidance, recovering part of the accuracy lost under aggressive token pruning.
Big Tech
Ahmadreza Jeddi · Enming Zhang · Jasper Gerigk · Hakki Karaimer · Mozhgan Nasr Azadani · Jiayun Luo · +9 more
University of Toronto · Vector Institute · Samsung AI Center Toronto · Stanford University · University of Waterloo
Research Digest··2 min read
Thread:Token Pruning Efficiency
In one line
Visual token pruning degrades VLM performance primarily due to unreliable use of remaining visual evidence, not irreversible information loss.
Before you read this
Three ideas this paper builds onon-policy distillationA training method where a student model learns from a teacher using its own current policy's outputs, often improving reasoning and instruction following.131 papers →privileged informationInformation available only during training that improves performance but is not accessible at inference.85 papers →context compactionCompressing or summarizing an LLM's past context into a smaller representation to extend memory and improve reasoning within limited token budgets.81 papers →
Read it as
Jeddi et al. investigate why reasoning vision-language models falter when most visual tokens are removed.
The authors first performed a fixed-context Pass@K analysis.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.