The authors first decomposed existing 4-bit NVFP4 training recipes into forward-pass choices, which affect predictions and gradients, and backward-pass choices, which determine how gradients are represented or estimated.
New scaling law better predicts losses from low-precision pretraining
The authors separately model quantization’s effects on model capacity, data efficiency and persistent optimization error across forward and backward passes.
Big Tech
Jinuk Kim · Rajarshi Saha · Muhammed Fatih Balin · Youngsuk Park
Seoul National University · Amazon Annapurna Labs
Research Digest··3 min read
Kim and colleagues develop a scaling law for predicting language-model pretraining loss when numerical precision varies between computational passes or Transformer layers.
Why this paper
From Amazon Annapurna Labs and Seoul National University
In one line
A scaling law for mixed-precision quantized pretraining achieves up to 11x lower loss prediction error than prior laws.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§