are trained on many graphics processors that constantly exchange data, and this communication can take longer than the math. Sending fewer bits per value helps, but extreme formats lose accuracy. FP4Quant sends values in a 4-bit floating-point format. Tiny clusters of values share scaling factors, preserving local patterns; values are packed two per byte; a fast transform keeps errors low. It was tested on a small model over several H100 GPUs. It improved end-to-end training throughput by up to 1.56x over standard 16-bit precision while keeping convergence comparable. The catch is a trade-off: training runs faster, but final model quality may drop.
4-bit compression speeds up LLM training by 56 percent with minimal accuracy loss
FP4Quant packs FP4 values efficiently on GPUs to reduce communication bottlenecks in tensor parallelism.
Research Lab
Man Liu · Xingchen Liu · Hairui Zhao · Guangming Tan · Dingwen Tao
University of Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences
Research Digest··2 min read
In one line
FP4Quant compresses model-parallel LLM communication to 4-bit floats, cutting quantization overhead and improving end-to-end training throughput by up to 1.56x.
Before you read this
Three ideas this paper builds onlarge language modelA neural network trained on massive text corpora to generate coherent text and perform diverse language tasks.259 papers →post-training quantizationQuantizing pre-trained neural network weights and activations to44 papers →quantization-aware trainingTraining a neural network with simulated quantization in the forward pass so its weights and activations adapt to low-precision formats, improving deployment accuracy.18 papers →
Read it as
The authors present FP4Quant, a framework that compresses communicated tensors to 4-bit floating-point numbers during model-parallel training of large language models.
Communication between GPUs, especially the AllReduce operations in tensor parallelism, becomes a major bottleneck as language models grow larger.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.