Training a language model after its initial pre-training, called post-training, often relies on a technique called policy optimization, where the model learns from rewards to improve its responses. A common method, REINFORCE, can fail by making the model less and less likely to produce the best answers over time, even as its average reward improves. This is called coverage collapse and wastes training steps.
The researchers propose a fix called Approximate Policy Mirror Descent (APMD). Instead of keeping the original reference policy fixed, APMD periodically resets the reference to the latest version of the model. This keeps the model close to a good baseline, preventing coverage collapse and making training much more efficient.
The main finding is that APMD guarantees the model can learn near-optimal responses using a number of samples that depends on the base model's quality. This explains why practical algorithms like PPO and GRPO work well when they include reference resets and strong regularization. For engineers, this means tuning the reset frequency and regularization together is key to reliable post-training.