Large language models often use a design called Mixture-of-Experts, where only a few specialized parts (experts) handle each piece of text. This keeps computation low per token, but on TPU hardware, moving data between memory and compute units becomes a bottleneck, especially when some experts are overloaded with work. Uneven workloads can make fast processing difficult.
This paper introduces Fuse-All MoE, a new kernel (a low-level program) that combines many steps of expert processing into one operation. It keeps intermediate data on the chip, reduces data movement, and uses scheduling tricks and expert duplication to balance the load across TPU chips, even when traffic is uneven.
In tests on Google TPUs, the new kernel reduced compute time for unbalanced workloads from 2.3x to just 1.09x of a balanced baseline. For smaller token batches (decode), it cut latency by up to 21.3%, improving overall server throughput by up to 34.5%. This makes MoE inference more efficient on today's hardware.
