The authors collected agent decision states from simulators with exact ground truth (ALFWorld, BabyAI-Text, TextWorld), customer-service policies (tau^2-bench), tool use, web navigation and coding traces.
Fast AI agents read states correctly yet still choose wrong actions
Most decision errors in System One models occur when all probed facts about the agent's state are answered correctly, and these errors cluster on the same states across systems.
Top University
Yubo Li · Ramayya Krishnan · Rema Padman
Carnegie Mellon University
Research Digest··3 min read
The authors built a frozen benchmark of 65,228 agent decision states and paired each next-action question with independent factual probes about the same state.
§