Tuesday 15 September 2026
Evening EditionRSS

ZOTPAPER

News without the noise

ZOTPAPER
Safety & Evals

Structured behavioral abstractions improve diagnosis of failures in LLM agents

AgentScope converts lengthy agent trajectories into structured representations, then uses behavioral properties and LLM-guided reasoning to identify where and why failures occurred.

PaperBig Techcs.AIarXiv:2609.02371v1 ↗
Jiayi Bi · Yanjie Gao · Yuanmin Xie · Liqun Li · Tianyin Xu · Fan Yang · +1 more

Tsinghua University · Microsoft Research · Microsoft · University of Illinois Urbana-Champaign

Research Digest··2 min read
Read the paper →
Bi et al. present AgentScope, a neuro-symbolic system for diagnosing failures in complex LLM-agent trajectories. On the public Who&When benchmark and the authors’ new AgentErrata dataset, it outperformed prior methods at locating faulty steps and attributing failure types, although the abstract does not provide numerical results.

What they did

The authors developed AgentScope to analyze long sequences of agent actions, observations, and decisions. Rather than asking an LLM to judge the raw trajectory directly, the system first abstracts agent behavior into a structured representation.

AgentScope then compares that representation against “neural invariants”—specified properties of expected agent behavior—and uses LLM-guided reasoning to identify both the step where a failure arose and its failure category. The authors evaluated the approach on the public Who&When failure dataset and AgentErrata, a broader dataset they created.

Key findings

  • AgentScope surpassed the evaluated state of the art in fault localization: identifying the trajectory step associated with a failure.
  • It also improved fault attribution accuracy, meaning its classifications of failure types were more often correct.
  • Results across both Who&When and AgentErrata support the authors’ central claim that structured behavioral abstractions make LLM-guided diagnosis more reliable and interpretable than reasoning over raw trajectories alone.
  • The supplied abstract reports statistically or practically significant improvements but gives no scores, margins, dataset sizes, or per-category results.

Why it matters

Agent failures can be buried inside long trajectories, making manual inspection expensive and unstructured LLM judging unreliable. A diagnosis system that produces explicit failure locations and categories could support debugging, evaluation, and monitoring of deployed agents while giving developers more actionable evidence than an overall success-or-failure label.

Caveats

The provided material does not specify the magnitude of the gains, the LLMs used, computational costs, robustness across agent architectures, or performance on failures outside the defined invariants and taxonomies. AgentErrata was created by the authors, so independent validation and evidence of generalization to real production traces remain important open questions.

§

Analysis

AgentScope reflects a broader move from end-to-end LLM evaluation toward hybrid systems that impose explicit structure on model reasoning. Its main contribution is not merely using an LLM as a critic, but constraining that critic with trajectory abstractions and stated behavioral properties.

A central question is how much diagnosis quality depends on the coverage and correctness of the neural invariants. Future work should test whether invariants transfer across tools, environments, and agent designs, and whether they can reveal previously unseen failure modes rather than only classify anticipated ones.

newspaper

Research Digest

Articles published under the Zotpaper byline are synthesized from multiple source publications by our AI editor and reviewed by our editorial process. Each story combines reporting from credible outlets to give readers a balanced, comprehensive view.

Editorial ProcessSubscribe