The paper introduces AGNI, an automated pipeline that creates novelty tasks for LLM agents by identifying assumptions in successful trajectories and injecting targeted environmental changes that invalidate those assumptions. This allows controlled evaluation of how agents adapt when the software environment changes while the task goal remains fixed.
Evaluating multiple LLM agents on three terminal benchmarks reveals a significant adaptation gap: agents perform worse on novel tasks compared to base tasks. Trajectory analysis shows that agents often detect the change but fail to diagnose its cause and revise their strategy. However, post-training agents on environmental novelties improves their adaptation to new, unseen novel tasks and even boosts performance on original base tasks, suggesting that environmental variation is a valuable dimension for agent training.