Cyber agents hide critical weaknesses behind successful attack workflows

The paper presents an end-to-end diagnostic study of an Autonomous Adversary system that uses orchestrator, executor, and validator LLMs to perform lateral-movement attacks in an enterprise-like environment. The authors evaluate six frontier models across two scenarios and three modes, introducing a cost-aware score to measure abnormal token use, retries, and runtime. They find that validators are generally relevant and evidence-grounded but often nonspecific and overly optimistic, and that bottlenecks cluster in credential and lateral-movement tasks, spreading with scenario complexity and varying more under full autonomy. This is interesting because it shows that focusing solely on success rates hides inefficiency, adaptation through retries, and poor state interpretation, and it provides a framework for localizing where progress becomes constrained in multi-step cyber agents.