The authors propose Exploration-Preserving Policy Optimization (ExPPO), a method that modifies the advantage computation in group-relative reinforcement learning with verifiable rewards (RLVR).
Simple credit shaping boosts exploration in language model reinforcement learning
ExPPO redistributes group-relative advantages based on response surprisal and pass rate, improving coverage and diversity without additional computation
Big Tech
Hangzhan jin · Mohammad Hamdaqa · Doina Precup
Mila - Quebec AI Institute · Polytechnique Montréal · McGill University · Google DeepMind
Research Digest··2 min read
The authors introduce ExPPO, a lightweight modification to GRPO that reshapes advantages using prompt-relative, length-normalized response surprisal and prompt pass rate.
Why this paper
From Google DeepMind and 3 others
In one line
ExPPO redistributes reinforcement-learning credit toward surprising responses, improving reasoning coverage and correct-answer diversity while preserving verifier reward direction.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§