Robot policies often need memory of past observations, but conditioning on full histories can hurt generalization, and VLM-based history summarization at deployment is slow and costly. This work shifts that expense to training: a VLM provides saliency labels for what matters in past frames, and a compact workspace token is trained to reproduce that salient information via a set-reconstruction decoder loss. At deployment, the token replaces raw history observations and is queried without running a VLM in the loop. The authors report that this works in both simulation and hardware, and that workspace tokens are not only lighter but also yield better task performance than baselines, including VLM-based frame selection.
Saliency-driven workspace tokens give robots lightweight task memory
Authors show that distilling vision-language model queries into a compact latent token during training enables memory-intensive manipulation without in-the-loop VLM reasoning.
Top University
Nitish Dashora · Douglas Chen · Idan Shenfeld · John Marangola · Pulkit Agrawal · Max Simchowitz
Massachusetts Institute of Technology · Carnegie Mellon University
Research Digest··1 min read
Thread:Memory Management for Agents
In one line
A lightweight latent memory, the workspace token, can replace VLM-in-the-loop history reasoning in robot policies, improving speed and task performance.
Before you read this
Three ideas this paper builds onagent memoryA repository where an agent stores past interactions and outcomes to inform future decisions.232 papers →vision-language modelA model that handles tasks combining visual and textual information, such as editing images from conversation or reasoning across images and text.251 papers →context compactionCompressing or summarizing an LLM's past context into a smaller representation to extend memory and improve reasoning within limited token budgets.79 papers →
Read it as
Dashora et al. propose the workspace token, a lightweight latent memory for robotic manipulation trained by using a VLM during training to identify task-salient historical information, then distilling it via set reconstruction.
The authors address the challenge of long-horizon robotic manipulation requiring memory of past states and actions.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.