MaskCoFT is a fine-tuning method that trains both routers and experts in mixture-of-experts (MoE) models together. During fine-tuning, a learnable binary mask restricts each layer's Top-K routing to a shared subset of experts, and those experts adapt to the tokens now directed to them. At inference, the mask becomes a soft prior that re-ranks experts, so all experts remain selectable but the concentration set is favored.
In simulations with a GPU cache of 4 experts per layer for Mixtral-8x7B and 12 for DeepSeek-V2-Lite, MaskCoFT reduced expert fetches per token by 23.7% and 10.1% relative to the base model. In real offloading systems, time per output token dropped by up to 16.4% and 5.5%, respectively. Average accuracy over nine benchmarks stayed above the base model by 0.92 and 0.53 points.
This work is interesting because expert loading is the dominant bottleneck in offloaded MoE inference. Prior caching and prefetching are limited by routing patterns, and router-only fine-tuning cannot adapt experts to new routing. MaskCoFT increases expert reuse without accuracy loss by co-adapting routers and experts, using only cross-entropy loss and no auxiliary losses.