Sharing random numbers speeds integer training when rounding errors are corrected

The authors report that structured stochastic rounding with error correction matches independent rounding’s validation loss while accelerating dense language-model training on TPU v5p.

Big Tech

Zeyu Wang · Yiheng Duan · Tao Chen · Donghyun Koo · Sihan Liu · Wenbo Zhang · +2 more

Google

Research Digest·28 October 2026·2 min read

Illustration: Sharing random numbers speeds integer training when rounding errors are corrected

In one line

S3R-EC matches independent stochastic rounding's quality in LLM pre-training while accelerating INT8 training by 1.11x.

Before you read this

An idea this paper builds on

quantization-aware trainingTraining a neural network with simulated quantization in the forward pass so its weights and activations adapt to low-precision formats, improving deployment accuracy.16 papers →

Read it as

Training AI models often uses lower-precision numbers to save time, but very small gradients get lost when rounding to the nearest value. Stochastic rounding (SR) prevents this by randomly choosing to round up or down, keeping the average correct. However, standard SR needs a separate random number for every value, which is slow.

The authors propose Structured Subspace Stochastic Rounding (S3R), which shares one random number across many values (one per token). This eliminates the slowness but creates a small error. Their correction, S3R-EC, subtracts the average error, restoring accuracy.

In large language model pre-training, S3R-EC matched the validation loss of standard SR exactly. It made training steps about 1.11 times faster than standard SR. This matters because it allows faster low-precision training without any quality loss.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

stochastic rounding shared random thresholds structured error correction quantization-aware training gradient error covariance