What they did

LoopArena fixes the coding agent, called the Worker, and evaluates a separate Controller responsible for choosing the next action after each coding round. The Controller receives a structured run summary and can assign further work, request verification, or stop the process.

The benchmark has three settings. Type I tests next-step control decisions using execution-validated questions without running the Worker during evaluation; Type II repeatedly controls a selected portion of a task; and Type III runs the complete task from its original state.

Key findings

• The highest observed Strict Success Rate on full Type III tasks was 24.69%.
• Across controllers, the paired lower-cost evaluation reduced estimated inference cost by an average of 64.4%.
• Type II closely matched the Type III controller ordering under the main Core criterion, with Spearman’s ρ = 0.9747.
• The results indicate that strong coding execution alone does not ensure reliable long-horizon control, verification, budget allocation, or stopping decisions.

Why it matters

LoopArena provides a way to evaluate agent orchestration separately from coding ability. Its partial-task setting may make comparisons of controller models substantially cheaper while preserving the ranking obtained from full end-to-end runs.

Caveats

The reported full-task success rate remains low, and the abstract does not specify how broadly the tasks, worker model, or controller set represent real software-development workflows. Because the benchmark fixes the Worker, its rankings may also depend on how particular controllers interact with that worker and with the benchmark’s structured summaries.