Large language models often use a technique called Mixture-of-Experts (MoE), which adds many extra sets of parameters (experts) but only uses a few per input. This should save computation, but on typical hardware the cost of moving all those extra parameters around cancels the savings, causing little or no speedup as models grow.
The authors rebuilt the training software stack in Olmo-core. They replaced a common parallelization method (Fully Sharded Data Parallel) with Distributed Data Parallel and added Expert Parallelism, which keeps each expert’s parameters on one GPU and routes only the relevant data to it. They also used MXFP8, a compact 8-bit number format, to reduce data movement and memory.
With this stack, they trained models from 12.9 billion to 1.2 trillion total parameters on up to 512 GPUs, achieving 858 trillion operations per second per GPU. This means MoE models can now be trained efficiently at very large scales, potentially leading to more powerful language models without proportionally higher hardware costs.
