LLM agents often fail in complex, multi-step ways that are hard to diagnose manually or with traditional bug-finding tools. Simply asking an LLM to judge the cause is unreliable, with even the best model achieving only 18.15% accuracy in attribution tests. AGENTSCOPE addresses this by first converting an agent's trajectory into a structured Reasoning-Action Graph (ReAG), then checking it against formalized neural invariants using LLM-guided reasoning.
This approach pinpoints both the failure step and its type. Evaluated on the Who&When dataset and a new, more comprehensive dataset (AgentErrata), AGENTSCOPE significantly outperformed state-of-the-art methods in localization and attribution accuracy. For example, on the Who&When Algorithm-Generated subset, localization accuracy ranged from 25.40% to 77.78%.