The authors developed a scaling law for reward optimization that jointly depends on the amount of preference data (M comparisons) used to train the proxy reward model and the KL divergence budget (K) allowed during policy optimization. They provide a theoretical derivation based on an information-theoretic model and show that performance scales roughly as the square root of the minimum of log M and K.
Using a real-world annotation setup with a 70B gold reward model and smaller proxy models (0.6B to 4B), they empirically validate this scaling law with R² values between 97% and 99%. The law holds across different optimization procedures (best-of-N and policy tilting), model sizes, and noisy preferences. This is interesting because it quantifies how increasing preference data yields diminishing logarithmic returns and offers guidance on setting divergence budgets relative to data availability.