LoopArena is a that isolates the ability of a model to act as a runtime controller for a separate (the Worker). It tests the Controller in three settings: Type I scores next-step contract selection without full execution; Type II evaluates control over a partial task slice; Type III runs the full task. The authors find that even the best Controller achieves only 24.69% strict success rate on full tasks, indicating substantial room for improvement. The benchmark also shows that the ordering of Controllers is reliable across settings (Spearman's ρ=0.9747) and that using a Controller can reduce estimated inference cost by 64.4% on average.
Models remain weak at steering coding agents through long tasks
LoopArena separates a controller model’s guidance from a fixed coding agent’s execution across three evaluation settings.
Chinese Tech
Yi Wang · Haopeng Zhang · Chengxiang Huang · Rui Dai · Kaikui Liu · Piotr Koniusz · +1 more
DreamX Team · Alibaba Group · Beijing University of Posts and Telecommunications · UNSW Sydney · Data61, CSIRO
Research Digest··2 min read
In one line
LoopArena evaluates how well a model can guide a separate coding agent, with best strict success rate only 24.69%.
Before you read this
Three ideas this paper builds oncoding agentAn AI-powered software system that autonomously writes, modifies, or extends source code based on high-level instructions.134 papers →benchmarkA standard set of tasks or data used to evaluate and compare the performance of AI systems.279 papers →agent harnessA system that structures, verifies, monitors, or controls how an AI agent interacts with external tools, environments, or execution runs.150 papers →
Read it as
The authors introduce a benchmark in which a controller model repeatedly reviews structured progress summaries and directs a separate coding worker, including deciding what to verify and when to stop.
LoopArena fixes the , called the Worker, and evaluates a separate Controller responsible for choosing the next action after each coding round.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.
What it released
Code