When training an AI agent to perform tasks that require many steps, a common method is to let it practice with a reward for finishing the whole task, known as multi-turn reinforcement learning (RL). This approach can improve long and complex tasks, but it is unclear exactly what the agent learns to do better.
The researchers tested agents after this training by giving them different numbers of allowed steps. They found that the benefit of multi-turn RL nearly disappeared when the allowed steps were limited. They also examined the quality of the agent’s decisions at each individual step and found that the training did not make every step better.
The main finding is that multi-turn RL mainly improves the agent’s ability to recover from mistakes and keep going on long tasks, rather than making every decision more accurate. This matters because in real-world use, where mistakes are costly or step limits are tight, such agents may fail when expected to perform well.