The paper introduces EvoPathBench, a benchmark that tracks how individual capabilities of self-evolving agents change over time as artifacts like memories or skills are updated. It evaluates three key capabilities: generalization to unseen tasks, retention after unrelated learning, and rule adaptation to new evidence. The experiments use public trading data and calibrated trajectories, freezing artifacts at successive checkpoints for evaluation on held-out episodes.
The results reveal that gains on similar unseen tasks often weaken under distribution shift, retention losses are concentrated in a minority of evolution paths, and no method achieves reliable rule adaptation. Additionally, while self-evolution can generate candidate artifacts with substantial held-out gains, the selected updates consistently fall short of realizing that potential. These findings highlight the need for and identify candidate evaluation and selection as critical areas for improvement.