Coding harnesses are usually evaluated as monoliths, so it is unclear which components drive agent performance. The authors built a lightweight harness with a fixed execution loop and varied three components: planning, action space, and context management. They ran 176 matched settings across four models on SWE-Bench Verified and Terminal-Bench 2.1, manipulating five context-management strategies and four context-window budgets, plus ablations of planning and action space.
They found that context management matters most when the context window is small, mainly by preventing context-overflow failures, and that rule-based elision followed by LLM summarization gives the best efficiency; making elided content recoverable adds machinery models rarely use. Planning acts as an accuracy scaffold for weaker models but becomes a cost saver for stronger models. Predefined tools help models with weaker bash skills, while bash-only interfaces let capable models operate effectively at lower cost, especially on command-line-centric tasks.
This is interesting because it moves harness evaluation from whole-system comparisons to component-level effects, and it shows that the value of each component depends on model capability, task type, and context budget.