AI agents learn more from human diagnosis than repeated self-review

Across three models on a workplace-task benchmark, one round of human failure diagnosis outperformed four rounds of agent self-judgment, while prescribed memory structures hurt performance.

Big Tech

Peizhi Niu · Ge Shi · Vaibhav Gorde · Sarthak Ghosh · Christos Faloutsos · Min Cheng · +1 more

University of Illinois Urbana-Champaign · Amazon

Research Digest·4 November 2026·2 min read

Illustration: AI agents learn more from human diagnosis than repeated self-review

In one line

Agents that organize their own procedural memory as linked wiki pages, with humans diagnosing failures rather than designing schemas, pass more enterprise tasks than baselines while storing less.

Before you read this

Three ideas this paper builds on

memory consolidationSummarizing or structuring past information to reduce token usage while preserving task accuracy.37 papers →agent memoryA repository where an agent stores past interactions and outcomes to inform future decisions.212 papers →continual learningA learning paradigm where a model sequentially acquires knowledge from multiple tasks without significantly forgetting earlier ones.54 papers →

Read it as

Agents built on large language models (LLMs) run multi-step workplace tasks through software tools, but their knowledge is fixed before deployment: company rules and unusual system states surface only while running and are lost afterwards.

Procedural memory is notes on when a procedure applies, how to carry it out and how to recover when it fails. The agent instead writes and links its own wiki pages from run logs and feedback, with no human-designed structure.

Pass rates on an enterprise benchmark rose 4.6 to 6.1 percentage points over the best comparison with far less storage, and one round of human diagnosis beat several rounds of agent self-judgment. Human effort pays off in explaining failures, not prescribing how knowledge is filed.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

memory consolidation agent memory continual learning failure attribution agentic scaffolding