The authors present ALTO, an open-source training recipe for the GPT-OSS-20B MoE model on the MLPerf Small MoE training task.
ALTO recipe closes MXFP4 training gap to BF16 for 20B MoE LLM
Combining hybrid block quantization, randomized Hadamard transforms, stochastic rounding, and weight de-oscillation yields validation loss within 0.007 of the BF16 baseline on the MLPerf Small MoE benchmark.
Industry
Han Wang · Zhitao Wang · Natalia Frumkin · Xuanwu Yin · Alireza Khodamoradi · Dong Li · +1 more
Advanced Micro Devices, Inc.
Research Digest··3 min read
The authors trained the 20-billion-parameter mixture-of-experts model GPT-OSS-20B under the MXFP4 4-bit microscaling format on the MLPerf Small MoE benchmark.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.