Self-improving agents aim to get better by modifying their own methods. But improvements may not transfer to new or harder problems.
Researchers built SelfImprove-Bench, a test suite with a generalization ladder: levels from repeating old tasks to solving new task types across skills like planning, reasoning, and exploration. Tasks are generated from templates for scalability.
Testing several self-improvement methods showed large differences across the ladder. The best method on easy problems often performed worst on hard ones, and rankings reversed. This reveals that current methods lack robust generalization, highlighting the need for structured evaluation.