The authors built OSWorld-Science, an integrated framework combining scientifically meaningful tasks, artifact-based evaluation, and a flexible agent harness.
New benchmark tests AI agents on real scientific software workflows
OSWorld-Science evaluates computer-using agents across molecular drawing, pathology analysis, statistics, and simulation, finding frontier models still struggle.
Top University
Dingyuan Dai · Heli Qi · Lei Liu · Yinxi Li · Baiding Chen · Zijun Dou · +25 more
Tsinghua University · University of California, Los Angeles · RIKEN AIP · University of Waterloo · Carnegie Mellon University
Research Digest··2 min read
Dai et al.
Why this paper
From RIKEN AIP and 15 others · Part of Process Agent Benchmarks, now 17 papers
In one line
Current VLMs achieve under 70% success on scientific software tasks, revealing major gaps in specialized interface use.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§