Long-horizon agentic RL suffers from because only final rewards are available. MileGPO gets process-level credit from grouped on-policy rollouts without training auxiliary models or doing extra environment interaction. It discovers candidate milestones on successful rollouts and recurring traps on failed ones, then weights them by confidence and checks them against local alternatives.
The three components are Milestone Discovery, Reliability-Calibrated Shaping, and Progress-Contrastive Calibration. Together they assign credit to intermediate behaviors that look like meaningful progress, while down-weighting uncertain candidates and testing whether a candidate's incoming transition beats observed transitions from the same state.
On ALFWorld and WebShop, MileGPO achieves state-of-the-art performance, and on ALFWorld the gap between in-distribution and out-of-distribution performance is small. That matters because better intermediate credit can make long-horizon LLM agents learn more efficiently from sparse reward.