Training an AI agent to complete long tasks, like booking a trip with many steps, is hard because it’s not clear which individual actions led to success or failure. Current methods compare actions that happen in the same situation, but they only use data from the same task, which limits their accuracy.
BLPO solves this by grouping actions that serve the same purpose across different tasks, using simple rules to define what counts as progress. It pools experience and adjusts for differences in task difficulty, so a good action in one task can help train the agent in another, without needing extra trial runs.
On two benchmark tasks (ALFWorld and WebShop), BLPO improved task success for both a 1.5-billion- and 7-billion-parameter model. One setup saw a 2.48 percentage point gain over prior methods. Agents also learned faster and took fewer steps to finish tasks, with almost no extra computing cost.