What they did

The authors analyze GRPO, a reinforcement-learning method that scores each rollout relative to rewards within the same sampled group. They identify cases where this group-relative magnitude cannot distinguish successful reasoning from a lucky guess.

They then introduce SignBalance, which preserves whether verifier feedback is positive or negative but replaces group-dependent advantage magnitudes with a global scale. A per-class rescaling, excluded from gradient computation, restores a zero-mean balance between positive and negative examples. The method is evaluated across mathematics and search-agent benchmarks at multiple model scales.

Key findings

• The authors identify three sources of “spurious advantage”: tasks with small bounded answer sets, bounded sub-cases embedded within open-answer tasks, and search settings where many explored paths can converge on the same answer.
• In these settings, standard GRPO can reinforce guess-like behavior because accidental correct answers receive the same favorable advantage magnitude as reasoned answers.
• SignBalance reportedly matches GRPO on open-answer mathematics, where lucky-answer collisions are less central.
• It outperforms GRPO on bounded-answer mathematics and search-agent benchmarks, although the abstract provides no numerical effect sizes.

Why it matters

The result challenges the assumption that verifiable correctness is always an adequate learning signal: when correct answers can be reached by chance through many equivalent options or paths, an optimizer may strengthen the wrong behavior. A composition-independent advantage estimator could make reinforcement learning with verifiable rewards more reliable for multiple-choice-like tasks and tool-using search agents.

Caveats

The supplied abstract does not specify benchmark names, model sizes, baselines beyond GRPO, or quantitative gains, so the magnitude and generality of the improvements cannot be assessed here. It also remains unclear how SignBalance behaves with noisy verifiers, partially correct outputs, or tasks whose rewards are not binary; the announced code had not yet been released in the provided text.