Large language models often use Mixture-of-Experts, a design where only a handful of specialized subnetworks (experts) process each token, saving computation. Yet all experts must reside in memory, requiring large GPU capacity. Expert offloading moves some weights to CPU and fetches them on demand, but existing systems fail to scale with multiple GPUs.
AdaOffload dynamically decides where to run each expert, CPU or GPU, and how to split work across GPUs based on real-time workload and hardware. It observes expert usage and cache residency to adapt its strategy for both decode and prefill phases.
On three large open-weight models across five hardware platforms, AdaOffload boosted throughput by 1.9 to 15.5 times over the best previous systems. This speedup lets organizations with limited GPU resources serve frontier models themselves, improving data control and reducing costs.