The authors assembled 85 assignments requiring agents to navigate project folders containing PDFs, spreadsheets, drawings, correspondence and logs.
AI agents fall short on a benchmark of professional engineering
Across 85 engineering assignments, the strongest tested configuration averaged 50.6% despite using 27 million tokens and 63 minutes per task.
Big Tech
Rujun Han · Yufan Zhuang · Arijit Ray · Han Yu · Zoey CuiZhu · George Lee · +7 more
Google Cloud AI Research · Google
Research Digest··2 min read
Han and colleagues introduce PLE, a benchmark that asks AI agents to produce engineering deliverables rather than answer isolated technical questions.
§