Mixture-of-experts models suffer from expert-weight traffic bottlenecks during batched decoding. SlimWise addresses this by applying only to the decode phase, while retaining the full expert pool for prefill. This decoupling allows the pruned decoder to reuse full-model KV caches without conversion, recovering much of the accuracy lost to pruning. Implemented in vLLM, SlimWise achieves up to 1.81x decode throughput improvement at 50% expert pruning with minimal accuracy loss, and supports both PD-disaggregated and PD-colocated serving.
Expert pruning in MoE models can be tailored to decode phase, preserving prefill quality while boosting throughput
SlimWise framework keeps full expert pool for prefill, prunes for decode, and reuses KV cache without conversion, achieving up to 1.81x decode throughput on Qwen3.6-35B-A3B with 50% expert pruning.
Top University
Gunho Park · Kyoungho Jeun · Juntaek Oh · Byeongjun Shin · Baeseong Park · Minsoo Rhu
a2sys · KAIST
Research Digest··3 min read
In one line
SlimWise improves MoE decode throughput by pruning experts only during decode while keeping full prefill.
Before you read this
Three ideas this paper builds onmixture of expertsA neural architecture with many specialized sub-networks, only a few activated per input via a router, balancing capacity and efficiency.91 papers →expert pruningSelectively removing or deactivating experts in a mixture-of-experts model to improve efficiency while preserving accuracy.3 papers →prefill-decode disaggregationSplitting LLM serving into separate prefill and decode phases to optimize GPU resource allocation and batch sizing.7 papers →
Read it as
The authors present SlimWise, a serving framework for mixture-of-experts (MoE) models that decouples expert pruning across prefill and decode phases.
The authors designed SlimWise to allocate expert capacity separately to the two inference phases of MoE models: prefill uses the full expert pool, while decode uses a pruned pool.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.