Standard Mixture-of-Experts (MoE) routing enforces uniform token allocation across experts, a design inherited from language models. This uniformity trap poorly suits video data, where tokens are spatiotemporally redundant and semantically long-tailed, causing coherent patches to be scattered across experts and leading to structural distortion in generated videos.
SplitMoE addresses this by splitting the expert pool into two roles: semantic experts that capture high-level semantic abstractions and generic experts that preserve residual visual details. Prototype-guided routing and pull-push regularization allow tokens to cluster by semantic attributes rather than forced uniformity, improving routing coherence and expert specialization.
Under matched activated-parameter budgets, SplitMoE outperforms standard MoE in convergence speed and video generation quality on standard benchmarks. It also reveals an emergent coarse-to-fine denoising pattern where semantic experts dominate early steps and generic experts take over later, suggesting a modality-aware scaling path for video .