The paper introduces ASPIRE, a for evaluating large language models' ability to self-evolve from vague natural-language capability goals, such as 'become a better researcher.' Unlike prior work that starts with explicit tasks and metrics, ASPIRE hides evaluation tasks and requires agents to interpret the goal, select training data, design learning signals, and decide when to stop. Experiments across six goals and 520 hidden items show that current agents routinely complete training and harness-editing loops, but weight-level gains are sparse and unstable. Agents often train on mismatched data and rely on narrow self-evaluations, so local improvements fail to transfer to the hidden evaluation, and continued search can erase earlier progress.
Models struggle to improve themselves from vague capability goals
ASPIRE tests whether agents can translate broad goals into effective training or harness changes without seeing the downstream evaluation tasks.
Independent
Yuhao Wu · Jingyuan Zhang · Jiajun Shi · Yuxuan Zhang · Xinping Lei · Junting Zhou · +15 more
Research Digest··2 min read
Thread:Agent Self-Improvement
In one line
Self-evolution from vague goals leads to mismatched training data and non-transferable improvements.
Before you read this
Three ideas this paper builds onself-evolving agentAn agent that improves by reusing its own experience, memory, or self-generated feedback, with success depending on verification and design.80 papers →harness evolutionIteratively adapting an agent's environment, prompts, or evaluation setup to improve task performance and generalization.56 papers →agent harnessA system that structures, verifies, monitors, or controls how an AI agent interacts with external tools, environments, or execution runs.150 papers →
Read it as
Wu and colleagues introduce ASPIRE, an interactive benchmark in which agents must interpret a natural-language capability goal, choose how to improve, and evaluate their own progress.
ASPIRE gives an agent a broad capability goal while withholding the downstream evaluation tasks.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.