The Agent Error Dataset (AED) provides 50,228 error-diagnosis pairs from 9,961 tasks across 33 environments, 19 harness families, and 23 policy models. A five-stage pipeline collects natural failures, generates diagnoses and corrections, and validates them against recorded evidence. Using this dataset, first-proposal corrections raised verifier pass rates from 18.4% to 51.1% on 3,062 matched replay pairs. Full-diagnosis fine-tuning improved Qwen3-8B's exact-step agreement from 47.2% to 63.6%, outperforming a strong prompted baseline of 54.7%. This work demonstrates that structured failure data can substantially boost both diagnosis and repair capabilities in LLM agents.
Dataset turns failed agent traces into diagnosis and repair supervision
The authors compile 50,228 diagnosed failures and show that proposed corrections often outperform retrying the original action.
Industry
Kunlun Zhu · Xuyan Ye · Yibo Li · Cheng Qian · Beibin Li · Heng Ji
Apodex
Research Digest··2 min read
In one line
A dataset of 50,228 error-diagnosis pairs enables LLM agents to learn from failures, improving correction success and diagnosis accuracy.
Before you read this
Three ideas this paper builds onfailure attributionIdentifying which component or step in a system caused an observed failure, often using structured trace representations.146 papers →credit assignmentDetermining which actions or steps in a sequence contributed to a final outcome, often for reinforcement learning.190 papers →counterfactual reasoningComparing actual outcomes with imagined alternatives to estimate the effects of different actions or conditions.114 papers →
Read it as
Zhu et al. present the Agent Error Dataset, a collection that preserves failed agent trajectories, identifies likely decision errors, and proposes evidence-checked corrections.
The authors collected 50,228 error-diagnosis pairs from 9,961 tasks spanning 33 text-based environments, 19 agent harness families, and 23 policy models.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.