The paper introduces policy-space scaling: using a completed reinforcement learning training history to construct policies that outperform any checkpoint in that history, without further training or increased inference compute. The authors propose SURGE, which combines two checkpoints from the same RL run—a high-accuracy anchor and a competitive shorter-response donor—by spectrally decomposing their parameter updates from initialization. Crucially, the combination is determined from the weights alone without testing candidate policies. Evaluated on 1.5B parameter math reasoning models (DeepSeek and Nemotron) and a 7B coding model (OLMo), SURGE improves benchmark accuracy over both input checkpoints while using fewer reasoning tokens than the anchor. For example, on DeepSeek AIME24 it achieves 54.17% against a best measured native checkpoint of 50.83%, and on OLMo HumanEval+ 83.7% against 82.8%. This is interesting because it reveals that RL histories contain untapped capability, offering a new axis for scaling beyond extending training or allocating more inference compute.
RL training runs can yield stronger policies than their best checkpoints
The authors demonstrate that combining two checkpoints from the same run via spectral fusion improves accuracy on math and coding benchmarks without extra training or inference compute.
Independent
Bangji Yang · Jiajun Fan · Hongba Ma · Ruihan Guo · Ge Liu
Research Digest··3 min read
In one line
An RL training history can yield policies stronger than its best checkpoint without extra training or inference compute.
Before you read this
An idea this paper builds onRead it as
The authors show that a completed reinforcement learning (RL) training history contains structure that can be exploited to construct policies stronger than any checkpoint visited during training.
The authors formulate policy-space scaling, expanding the set of deployable policies from a fixed RL history without extending training or increasing inference compute.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.