Language models can lag 50 to 150 updates behind training

Gao and Gao report that rounding weights for training-data generation hides most individual updates, while unbiased rounding improves results across matched runs.

Top University

Andrew Gao · William Gao

Stanford University

Research Digest·2 November 2026·2 min read

Illustration: Language models can lag 50 to 150 updates behind training

In one line

Standard bf16 rounding silently discards most RL updates, creating a hidden lag that unbiased rounding eliminates.

Before you read this

Three ideas this paper builds on

post-training quantizationQuantizing pre-trained neural network weights and activations to39 papers →reinforcement learningA machine learning method where an agent learns to make sequences of decisions by receiving rewards or penalties for its actions.245 papers →non-determinismSame inputs give different outputs across runs, confounding testing and evaluation.27 papers →

Read it as

Training a language model with keeps a high-precision master copy of its weights for making updates but uses a lower-precision copy for generating practice examples. Standard round-to-nearest conversion of the master to the lower precision silently discards most small updates, creating a hidden lag where the data-generating model is tens of steps behind the one the optimizer thinks it is improving.

The fix replaces round-to-nearest with an unbiased method so no updates are systematically lost, and computes the training loss on the exact copy used for generating examples. This alignment, called Fair Cast, requires changing a single line in the weight export call.

Fair Cast led to faster and more stable learning in a majority of tests and prevented a collapse that consistently happened on long rollouts. It also makes the saved checkpoint agree with the served model, eliminating a discrepancy developers had been unable to detect.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

post-training quantization reinforcement learning non-determinism large language model group relative policy optimization quantization-aware training