What they did

The authors evaluated agents on 100 multi-file RefactorBench tasks, holding the execution environment fixed while varying model backbone, prompt specificity, and execution regime: retrieval-free baseline, retrieval-augmented single agent, or multiple agents.

RefactorPlatform runs each attempt in an isolated workspace and records tokens, code diffs, transcripts, and other telemetry. It supports OpenRouter models and GitHub Copilot CLI, streams terminal activity, and uses abstract syntax tree (AST)-based checks to verify refactorings.

Key findings

• AST-aware code chunking outperformed naive fixed token-window chunking by 25–30 percentage points across prompt modes.
• Naive retrieval performed worse than the retrieval-free baseline, indicating that adding context without structuring it can hinder an agent.
• On matched tasks, the retrieval-augmented single agent succeeded on 86%, compared with 66% for the evaluated multi-agent configuration. No task was solved only through delegation.
• Retrieval increased token use, but its higher accuracy left the cost per successful refactoring effectively unchanged.

Why it matters

Repository-scale coding evaluations often change models, prompts, retrieval systems, and orchestration simultaneously, making results difficult to interpret. RefactorPlatform provides a reproducible way to isolate those choices and suggests that careful context selection may matter more than adding agent complexity.

Caveats

The demonstration covers 100 tasks and one particular sub-agent design, so it does not establish that single-agent systems generally outperform multi-agent ones. Results may also vary with other repositories, refactoring types, retrieval methods, models, and verification criteria; broader testing is needed to establish generality.