Sandboxed computer environments let language models use tools and persist files, but extending them to vision-language models (VLMs) creates a state-management problem. VLMs produce intermediate visual evidence—crops, masks, overlays—that must stay addressable without overloading context. The authors introduce VLM-in-Sandbox, a training-free framework that separates evidence generation (via sandbox tools) from evidence management via a Visual Workspace. The workspace maintains an image ledger, a bounded active visual context with two slots (focus and aux), and requires the model to explicitly promote selected artifacts for later inspection.
Across seven benchmarks and four base VLMs, VLM-in-Sandbox achieves the highest sample-weighted average accuracy among vanilla, append-only sandbox, and the proposed method. A matched ablation on 1,260 examples shows 66.27% accuracy with 18.6% fewer total tokens than the automatic retain-all control. Over 6,350 GPT-4.1-mini examples, it produces 302 rescues and 142 regressions relative to original append-only. A local vLLM study confirms reduced uncached tokens, faster first token, and lower latency. These results identify explicit visual evidence state as a missing abstraction for sandboxed VLM agents.