for AI agents often measure task length by the number of steps the agent takes. But this count can be misleading: an agent might finish everything in its first turn by running one script, making a long-looking task actually short. The true challenge is depth: steps where the agent must read the output of one command before it can decide the next one.
The authors propose measuring depth by setting a maximum number of turns and seeing the smallest cap under which a task still passes. They audited 734 tasks from six existing datasets and found that 94% of solved synthetic tasks pass within three turns, meaning they add work but not depth. Depth, unlike turn count, predicts how hard models find a task after controlling for the number of operations.
They built 105 new tasks where each segment is generated in a real software environment, forcing later steps to depend on earlier ones. No current model passes half of these tasks, and fine-tuning a small model on runs from these tasks improved its performance on two standard benchmarks.