Process Agent Benchmarks
Benchmarks that decompose agent tasks into subgoal processes for fine-grained evaluation of agent performance and error analysis.
76 papers · 1 months
Where this stands
The written synthesis of this thread is for subscribers. Subscribe.
Results across this thread
3 reported results from the papers in this thread.
The table is for subscribers. Subscribe to see every reported number side by side.
How this thread developed
Sept 2026 · NVIDIA
Process-based evaluation reveals where computer-use agents go wrong
Decomposes computer-use tasks into subgoals for process-level evaluation, identifying failure points.
3 further papers
Oct 2026 · Northeastern University, Inclusion AI, Ant Group
Terminal agents verify often but miss and mishandle many errors
evaluates terminal agents' verification and error handling using TerminalBench2.1
1 further paper
Oct 2026 · Zhejiang University, Imperial College London
Clinical language models diagnose well but often mishandle full encounters
Introduces a clinical process benchmark that evaluates the steps of assessment, not just final diagnosis, advancing fine-grained agent evaluation.
Oct 2026 · Gradient Data, McGill University
Coding agents nearly finish repositories but still miss explicit requirements
Introduces Zero2Repo, a benchmark for repository-level coding agents with fine-grained failure analysis.
1 further paper
Oct 2026 · Rutgers University, Harvard University
LLMs fail multi-step math despite solving each step correctly
Introduces OracleLadder, a diagnostic tool that provides sub-goal roadmaps to evaluate multi-step math reasoning at the process level.
released code
Oct 2026 · University of Minnesota, Twin-Cities, IBM Research
Chart editing benchmark separates completed requests from unwanted code changes
Presents ChartRevise, a benchmark for chart editing via source code with reference-free evaluation, separating completed requests from unwanted changes.
Oct 2026 · Meta AI, Columbia University
Separating research choices from execution improves long-horizon agent performance
proposes a hierarchical agent that separates planning from execution into explicit subgoals
1 further paper
Oct 2026 · Renmin University of China, Tsinghua University
Outcome-only reinforcement learning can teach models to judge reasoning steps
Introduces a method that rewards outcome judgment while training step-level labeling, providing a new training signal for process benchmarks.
1 further paper
Oct 2026 · KAIST, Genentech
AutoSciBench automatically generates scientific agent benchmarks that adapt as agents improve.
Proposes an automatically generated and iteratively refined benchmark framework that adapts as agents improve, with process-level task decomposition.
2 further papers
Oct 2026 · JetBrains Research
Coding-agent benchmarks should mirror the task flows of their target users
Analyzes real coding agent sessions to argue benchmarks should mirror actual task flows, informing process decomposition in benchmarks.
1 further paper
Oct 2026 · ServiceNow Research, McGill University
New benchmark tests AI judges on long computer-use tasks by swapping instructions
Introduces AgentHorizon, a benchmark of instruction-trajectory pairs for evaluating AI judges on long computer-use tasks.
2 further papers
Oct 2026 · Chemnitz University of Technology, Fraunhofer Institute for Molecular Biology and Applied Ecology
Structural baselines mask real progress in agent failure detection
Shows that structural baselines (e.g., step position, run length) account for reported agent failure detection performance, questioning existing metrics.
5 further papers
Oct 2026
AI agents still cannot reliably deliver end-to-end on underspecified business requests.
Introduces a benchmark for evaluating agents on underspecified business requests requiring clarification and iterative optimization.
2 further papers
Oct 2026 · University of Texas at Austin, The Ohio State University
Frontier models struggle when correct actions must arrive on time
Introduces benchmark for agent tasks with temporal constraints, enabling fine-grained process evaluation.
Oct 2026 · Carnegie Mellon University
Fast AI agents read states correctly yet still choose wrong actions
Benchmark of 65k decision states analyzing gap between state reading and correct action.
2 further papers
Oct 2026 · Barcelona Supercomputing Center, University of Amsterdam
Formal plan validation shows LLM agents rarely plan as claimed
Adds PDDL-based mechanical validation of every action and goal to test whether agents actually plan as claimed rather than in natural language only.
5 further papers
Oct 2026
Benchmark tests LLM agents on recreating published computer science contributions
Proposes FrontierCS 2 for fine-grained evaluation of agents on recreating CS contributions.
1 further paper
Oct 2026 · University of Pennsylvania, University of Illinois at Urbana-Champaign
Agents write verifiers that wrongly reject valid solutions more often than they accept them
Adds a benchmark with hidden test cases showing agent-generated verifiers often wrongly reject valid solutions.
1 further paper
Oct 2026 · The Hong Kong University of Science and Technology, National University of Singapore
Dual Exam benchmarks agents and world models separately, exposing weak coupled performance
Introduces Dual Exam, a two-track benchmark that separately evaluates agent performance and world model accuracy, enabling fine-grained analysis of coupled weaknesses.
1 further paper
Oct 2026 · The Chinese University of Hong Kong (Shenzhen), ByteDance Inc.
Language models can understand others’ beliefs yet fail to act accordingly
Introduces a multi-level evaluation of theory of mind that separates belief inference from planning, enabling finer-grained assessment of agent social reasoning.
3 further papers
Oct 2026 · LogicStar.ai, ETH Zurich
Coding agents find real bugs, but rarely the ones maintainers fix
Introduces PROBE, a benchmark of real-world bugs that decomposes coding agent performance into recall and precision against maintainer fixes, enabling fine-grained error analysis.
3 further papers
Oct 2026 · Oracle AI
AI agents that succeed at tasks may still fail their mid-way steps
proposes trajectory-level evaluation protocol that checks subgoal process steps.
3 further papers
Nov 2026
AI agent benchmarks add work but not depth
Distinguishes parallel work from sequential depth and finds most benchmark tasks measure work rather than true reasoning depth.
Nov 2026 · Amazon
Coding agents' unnoticed mistakes make later requests fail more often
Evaluates coding agents on chains of requests from real projects, finding that unnoticed mistakes cause later failures, highlighting the need for fine-grained error tracking.
Nov 2026 · Adelaide University, CSIRO
Agent evaluation pipelines drop tool calls before the AI judge reads them
Reveals that evaluation pipelines drop tool calls before AI judges see them, criticising the reliability of process-level agent benchmarks.
4 further papers
Nov 2026 · Renmin University of China, Fudan University
Tool-centric method synthesizes harder long-horizon agent tasks from execution traces
Synthesizes long-horizon multi-tool tasks from real execution traces, enabling harder benchmark construction for agent process evaluation.
1 further paper
Nov 2026
AI agents fail end-to-end physics research due to shallow reasoning and missing intuition
Adds a benchmark of 56 end-to-end physics research tasks created by 42 physicists to evaluate agent research workflows.
3 further papers
Nov 2026 · University of Washington, National University of Singapore
AI agents benefit from checking both plans and imagined outcomes
Provides a shared benchmark across four domains for evaluating agent planning with imagined outcomes.
1 further paper
Nov 2026 · National University of Singapore, UC Riverside
AI agents turn binary patches into exploits, but only 20% succeed
Adds a benchmark that decomposes real-world CVE exploitation into subgoal processes, enabling fine-grained evaluation of agent performance across static and dynamic analysis steps
29 of 76 papers shown