AI agents are entering science, but current tests ask fixed questions or ask models to reproduce known papers. That does not show whether an agent can do open-ended research. The gives agents only a high-level prompt, like a principal investigator would give, and expects them to review literature, plan, run research, and write a manuscript. Results are graded by automated checks and expert rubrics. Across many frontier large language models (LLMs, the AI systems behind chatbots), the best configuration completed only 46 of 168 attempts. The main failures were insufficiently rigorous reasoning and limited expert intuition, not missing physics knowledge. So the next step is making agents more rigorous, not just more knowledgeable.
AI agents fail end-to-end physics research due to shallow reasoning and missing intuition
A benchmark of 56 research tasks shows top models produce publishable results less than a third of the time.
Independent
Bingran You · Wenjun Ke · Ria Rosenauer · Bo Liu · Wenbo Chen · Min Chen · +66 more
Research Digest··2 min read
In one line
An end-to-end physics research benchmark finds frontier AI agents fail most tasks, with main weaknesses insufficiently rigorous reasoning and limited expert intuition.
Before you read this
Three ideas this paper builds onagentic evaluationMeasuring AI agents' performance in complex tasks, highlighting failures in planning, monitoring, and real267 papers →benchmarkA standard set of tasks or data used to evaluate and compare the performance of AI systems.310 papers →verificationChecking outputs for correctness before accepting them, often to catch errors or reward hacking.297 papers →
Read it as
The authors introduce FrontierPhysics, a benchmark with 56 research tasks created by 42 physicists based on their own work.
The authors built FrontierPhysics, a of 56 end-to-end physics research tasks covering theoretical, computational, and experimental domains.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.