(GRPO) assigns each rollout an advantage magnitude based on within-group reward statistics. The authors show that this gives high magnitude to correct answers even when they are guessed, especially in bounded-answer tasks, open-answer tasks with small answer sets, and multi-turn search agents. They propose SIGNBALANCE, which uses a global scale and per-class rescaling to remove the spurious component from the advantage. Experiments show that SIGNBALANCE matches GRPO on open-answer math and improves on bounded-answer math and search agents.
GRPO Can Reward Lucky Guesses as If They Were Reasoning
The authors identify a bias in GRPO’s advantage estimator and propose SignBalance to reduce guess-like behavior while retaining performance on open-answer mathematics.
Big Tech
Jiamian Wang · Samyadeep Basu · Koustava Goswami · Tong Yu · Zhiqiang Tao
Rochester Institute of Technology · Adobe Research
Research Digest··2 min read
In one line
GRPO's advantage estimator rewards guessed correct answers, and SIGNBALANCE fixes this by decoupling magnitude from group statistics.
Before you read this
Three ideas this paper builds ongroup relative policy optimizationA reinforcement learning method that optimizes policy by computing advantages based on group-level reward statistics.154 papers →advantage estimationIn policy optimization, a method that calculates the relative value of actions using group statistics or baselines to guide learning.45 papers →reinforcement learning with verifiable rewardsReinforcement learning that uses rewards from objectively checkable signals like correct answers or verification feedback.138 papers →
Read it as
Wang et al. show that Group Relative Policy Optimization (GRPO) can assign large advantages to rollouts that reach correct answers by guessing rather than reasoning.
The authors analyze GRPO, a reinforcement-learning method that scores each rollout relative to rewards within the same sampled group.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.