DeskForge controls application state, content, window arrangement, stacking, appearance and display resolution.
Controlled desktop scenes improve visual grounding for computer-use agents
DeskForge generates densely annotated, configurable desktop scenes that improved four vision-language models on grounding benchmarks and multistep computer tasks.
Big Tech
A. Said Gurbuz · Ahmed Nassar · Sunghwan Hong · Marc Pollefeys · Peter W. J. Staar
ETH Zurich · IBM Research Zurich · Microsoft
Research Digest··2 min read
Gurbuz and colleagues built a Linux environment that composes real applications while collecting screenshots, accessibility data, window geometry and action outcomes.
Why this paper
From IBM Research Zurich and 2 others
In one line
Controllably composed real desktop scenes provide dense supervision that improves GUI grounding and long-horizon computer-use task completion.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (2 noted)
- ✓Reports numbers on named benchmarks (5 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§