What they did

The authors grouped on-policy trajectories and searched successful runs for recurring intermediate milestones and failed runs for recurring traps. Their reliability-calibrated shaping component weights these candidates according to how consistently they predict final outcomes.

A progress-contrastive calibration step then checks whether each candidate represents local progress and whether the transition leading to it performs better than observed alternatives from the same state. The method was evaluated on ALFWorld, a text-based household simulator, and WebShop, a simulated online-shopping benchmark.

Key findings

• The authors report state-of-the-art performance on both ALFWorld and WebShop, although the abstract does not provide scores or baseline comparisons.
• On ALFWorld, MileGPO reportedly maintains a small gap between in-distribution and out-of-distribution performance.
• Ablations indicate that reliability weighting, local-progress checks, and same-state branch comparisons provide complementary benefits.
• Credit diagnostics suggest that these components help distinguish useful intermediate progress from ambiguous or misleading states.

Why it matters

Long-horizon agents often receive only a final success or failure signal, making it difficult to determine which earlier actions deserve credit. MileGPO shows a way to derive denser process supervision from rollouts already collected during training, potentially improving learning efficiency without adding a separate evaluator or requiring more environment interaction.

Caveats

The evidence is limited to two simulated agent benchmarks, and the abstract gives no quantitative results, computational costs, statistical uncertainty, or details about baseline strength. It therefore remains unclear how well the method scales to substantially longer, noisier, or real-world tasks, and whether milestone inference remains reliable when few successful trajectories are available.