The paper addresses the problem of using LLM judges to automatically evaluate agent trajectories while guaranteeing a bounded error rate. Standard selective-judging methods assume trajectories are independent, but agent trajectories are clustered by task, leading to overconfident certificates. The authors introduce TaskBoot, a task-level bootstrap certificate that remains valid under clustering and provides practical coverage, unlike finite-sample cluster-valid alternatives that certify nothing at realistic task counts. Their method enables a 4B judge to certify 30-59% of evaluation on tool-use and web corpora at a 10% error budget.
Task-aware certificates safely automate part of LLM agent evaluation
Resampling at the task level let the authors certify automatic decisions despite correlations among agent trajectories.
Research Lab
Chengguang Gan · Yunhao Liang · Qinghao Zhang · Shiwen Ni
Independent Researcher · University of Chinese Academy of Sciences · Pusan National University · Shenzhen University of Advanced Technology
Research Digest··3 min read
In one line
A task-level bootstrap certificate ensures LLM judge error stays under budget, enabling safe selective automation of agent evaluation.
Before you read this
Two ideas this paper builds onRead it as
Gan et al. study how much human review an LLM judge can replace while keeping errors among automated decisions below a specified budget.
The authors treat agent evaluation as selective prediction: a judge automatically labels only trajectories for which it is sufficiently confident, leaving the rest to humans.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.