What they did

StarHarness evolves environment-specific components around a fixed language model, including task framing, prompts, tool interfaces, skills, MCP-backed providers, subagent organization and agent-loop configuration. It builds a compact search pool by grouping tasks according to how the baseline agent fails.

The framework separates tasks visible to the change proposer from hidden tasks used to select changes, then evaluates generalization on a further held-out set. The authors tested it on ITBench SRE, EnterpriseOps-Gym ITSM and AutomationBench Finance.

Key findings

• Evolved harnesses improved full-benchmark performance by 20–35 percentage points over the default harness.
• These gains required 4–12 accepted harness changes per environment.
• Improvements persisted on tasks excluded from evolution and transferred without further evolution across GPT and Qwen model families.
• Trace analysis associated the gains with repaired tool interfaces, encoded environment conventions and operational knowledge; several settings also showed fewer false-positive diagnoses and shorter trajectories.

Why it matters

The results suggest that persistent failures in tool-rich enterprise tasks can arise from mismatches between a capable model and its operating environment, rather than model weights alone. Systematically adapting the harness may therefore offer a lower-cost alternative to retraining while improving how agents navigate specialized tools and workflows.

Caveats

The reported evidence covers three benchmarks in site reliability engineering, IT service management and finance, not live production deployments. The abstract does not establish how evolution costs compare with alternative optimization methods, how stable the harnesses remain as tools change, or whether the approach generalizes beyond the tested GPT and Qwen model families.