The authors trained models from the Nemotron family, including a dense 8B model and a mixture-of-experts model with 30 billion total parameters and roughly 3 billion active per token.
Selective high-precision layers close the accuracy gap in four-bit training
Tests on dense and mixture-of-experts language models identify which quantization techniques remain effective over long training runs.
Big Tech
Anjulie Agrusa · Andrei Panferov · Elizabeth Wei · Keith Wyss · Paul Gibbons · Erik Schultheis · +4 more
NVIDIA · Institute of Science and Technology Austria · Red Hat AI
Research Digest··2 min read
Thread:Low-Precision Training
Agrusa and colleagues systematically compare recipes for training language models with NVFP4, a hardware-supported four-bit floating-point format, against conventional BF16 training.
§