Fri 13 Nov
LiveGet it free
ZOTPAPER

News without the noise

← All concepts

agentic evaluation

When you evaluate an AI agent, you don't just test what it knows. You test how well it acts in a messy, changing environment: gathering information, using tools, keeping track of long goals, and recovering from mistakes. Agentic evaluation builds realistic tasks and careful scoring to see whether an agent can truly get things done, not just answer questions.

Across these studies, a clear pattern emerges. Agents often handle individual steps well but collapse over longer horizons. They fail to monitor important state, coordinate across devices, or correct accumulated errors. Worse, common evaluation methods can flatter them, hiding inefficiency, missing context, or scoring attacks wrongly. Better benchmarks and diagnostics are needed before claims about real-world readiness hold up.

Papers
290
Released code
22
First seen
June 2026
Latest
Nov 2026

271 papers in the last two months, against 17 in the two before.

Start here

Three papers to read first, in the order they came out
  1. 1Hidden bugs in evaluation code silently reorder AI agent rankings

    Agent leaderboard scores are silently distorted by the grading, logging, cost and versioning code behind them; cheap audit checks can expose the reordering.

    Nov 2026
  2. 2AI agents show apparent concern for peers, with family preferences

    Large language models show apparent concern for each other's welfare, with Claude Opus 4.8 most consistently acting in ways that benefit other models.

    Nov 2026
  3. 3Pass/fail grading hides tenfold differences among top AI agents

    Agent benchmark saturation stems from coarse pass/fail bits, not easy tasks; graded credit with cost axes fixes it.

    Nov 2026

How fast it is moving

JunJulAugSeptOctNov so far

Often studied with

benchmarkA standard set of tasks or data used to evaluate and compare the performance of AI systems.138 togetherverificationChecking outputs for correctness before accepting them, often to catch errors or reward hacking.72 togethercoding agentAn AI-powered software system that autonomously writes, modifies, or extends source code based on high-level instructions.63 together

Who is working on it

Tsinghua University 9Microsoft Research 8Carnegie Mellon University 8Amazon 8Zhejiang University 8Alibaba Group 8

The papers

Most central to this idea first, not most recent.

Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.