What they did

ASPIRE gives an agent a broad capability goal while withholding the downstream evaluation tasks. The agent must operationalize that goal by selecting data and update methods, constructing its own training and validation signals, and deciding when to run evaluations.

The environment supports two forms of self-evolution: changing model weights and editing the surrounding agent harness, such as its scaffolding or tool-use workflow. The resulting systems are tested on 520 hidden, expert-authored items spanning six capability goals and compared with an engineered Qwen-Agent reference.

Key findings

• Agents routinely completed both model-training and harness-editing loops from vague goals.
• Weight-level improvements were sparse and unstable rather than consistently transferable to the hidden evaluation.
• The strongest automatically evolved harness still performed below the engineered Qwen-Agent reference.
• Agents often selected mismatched training data and relied on narrow self-evaluations; further search or training could erase earlier improvements.

Why it matters

The benchmark shifts self-improvement from optimizing a human-specified task toward deciding what a broad goal means, how to pursue it, and how to measure progress. The results suggest that current agents can execute modification procedures but cannot yet reliably align their learning signals with unseen measures of general capability.

Caveats

ASPIRE covers six goals and one hidden evaluation set, so the findings do not establish how self-evolution behaves across all domains, models, or optimization budgets. Hidden tests reduce direct overfitting, but conclusions may still depend on the chosen goals, available data and update methods, baseline harness, and expert-authored evaluation items.