Assigning credit to intermediate steps in multi-step reasoning tasks is a central challenge for training LLMs with sparse terminal rewards. Standard actor-critic methods like PPO learn value functions to construct token-level advantages, but reliable value estimation requires the critic to assess progress toward a correct solution and predict the policy's future behavior, which is difficult and can destabilize training.
πPPO addresses this by reusing verified same-prompt rollouts as contrastive evidence for the critic. For each target rollout, it conditions the value estimate on a leave-target-out context of verified sibling rollouts, preserving a valid baseline without additional generation beyond the group already collected for policy updates.
Experiments on mathematical reasoning benchmarks show that πPPO consistently improves value-estimation quality and outperforms representative actor-critic and critic-free RLVR baselines, even when paired with substantially smaller asymmetric critics. This offers a way to improve training stability and final policy performance without extra inference cost.