Large AI models that use mixture-of-experts (MoE) combine attention layers and expert layers, but running them on the same GPUs can be inefficient because attention and experts compete for memory. The researchers built FastAFD, a system that runs attention and experts on separate GPUs connected by rack-scale NVLink, a fast inter-GPU link. They compared it to wide expert parallelism, the standard approach where all GPUs run both layers but split experts across them. FastAFD achieved up to 1.17 times higher throughput for some models and matched the standard approach for others. The key insight is that the advantage depends on batch size benefits, overlap gains, and the cost of dedicating GPUs, helping operators decide when to use this approach.
MoE model disaggregation yields modest speedup on rack-scale NVLink, model predicts payoff
FastAFD engine achieves up to 1.17x throughput over wide expert parallelism on GB200 NVL72, with an analytical model that forecasts the gain within 3%.
Top University
Yichao Fu · Yuxuan Zhang · Minshen Zhang · Ruitian Wang · Junda Chen · Hao Zhang
University of California, Berkeley
Research Digest··3 min read
In one line
On rack-scale NVLink, attention-FFN disaggregation can deliver up to 1.17x more throughput than wide expert parallelism for some MoE models.
Before you read this
Two ideas this paper builds onmixture of expertsA neural architecture with many specialized sub-networks, only a few activated per input via a router, balancing capacity and efficiency.77 papers →speculative decodingA method that uses a lightweight model to propose token sequences that are then verified in parallel by a larger model.51 papers →
Read it as
The authors built FastAFD, an attention-FFN disaggregation (AFD) engine for GB200 NVL72, and benchmarked it against a well-tuned wide expert parallelism (wide EP) baseline.
The paper investigates whether attention-FFN disaggregation (AFD) improves decode throughput for mixture-of-experts (MoE) models on rack-scale NVLink systems like NVIDIA's GB200 NVL72.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.