When training large language models (LLMs) to solve reasoning problems using reinforcement learning, the model often stops improving early, a problem known as premature convergence. Some researchers attribute this to the model's outputs becoming too similar (low entropy), but simply making outputs more diverse does not reliably fix the plateau.
The researchers introduced "reward budgeting": for each question, they set a fixed total reward to be shared among all correct responses. When many responses are correct, each gets a smaller reward share, which automatically slows learning once a question is well-solved. This prevents the model from locking into narrow patterns prematurely.
On math reasoning benchmarks, reward budgeting improved performance by up to 12.50 points on AMC23 and 10.00 on AIME24. Notably, it achieved these gains without actively forcing diverse outputs, showing that premature convergence is not solely an entropy problem. This provides a practical way to get better results from reinforcement-learning training.