Automated harness optimization for LLM agents updates prompts, tools, and control logic from execution feedback, but typically uses a fixed set of training scenarios. ActiveSaddler introduces an that selects which scenarios to use based on observed failures, modeled as a non-stationary bandit with failure-pattern arms. It abstracts recurring failures into reusable arms, estimates their potential for further improvement, and balances revisiting known weaknesses with exploring unseen scenarios.
On GAIA2 and Terminal-Bench 2.0, ActiveSaddler improved test Pass@1 by 4.4 and 7.5 percentage points over the same optimizer using a fixed scenario order. Ablations confirmed that dynamically constructing optimization targets, estimating their evolving utility, and balancing continued optimization with new failure discovery are all essential for the gains.
This work establishes automated curriculum learning as a new dimension for harness optimization, showing that adapting the training curriculum alongside the harness itself yields significant improvements, especially under limited rollout budgets.