The authors build on Group-Relative Policy Optimization (GRPO), a reinforcement learning method for training reasoning models.
Turning failed reasoning attempts into training signal improves GRPO on hard problems
RSHD generates corrective notes from failures, verifies them via replays, and distills successful behaviors into the policy without extra inference cost.
Big Tech
Sheldon Yu · Tong Yu · Nikki Lijing Kuang · Zhendong Chu · Zhenwei Tang · Sungchul Kim · +3 more
University of California, San Diego · Adobe Research · University of Virginia · University of Toronto · University of New South Wales
Research Digest··3 min read
The authors propose Replay-Selected Hindsight Distillation (RSHD), which lets a reasoning model generate a corrective note after a failed attempt without seeing the answer, re-solve the problem with that note, and keep the replay only if it outperforms the original group.
Why this paper
From Adobe Research and 4 others
In one line
Failed reasoning attempts can be turned into verified supervision for GRPO by having the policy write corrective notes and distilling only replays that beat the original group.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§