The paper introduces MoRE, a modification to mixture-of-experts (MoE) layers that replaces the standard dense router with a low-rank factorization. This reduces the per-token routing cost from Θ(Mh) to O((h+M)r), where M is the number of experts, h is the hidden dimension, and r is the chosen rank. The authors prove that a rank logarithmic in M is sufficient to maintain the router's expressivity and load-balancing ability, meaning low rank does not hurt routing quality.
At matched active FLOPs, the factorization allows MoRE to support a factor of Θ(h/r) more experts than a standard MoE layer. To realize this gain in practice, the authors design a fused Triton kernel that keeps scores in on-chip SRAM, avoiding expensive HBM traffic. Empirical results show that MoRE improves memorization on a synthetic phonebook task and improves performance on knowledge-intensive Q&A benchmarks (TriviaQA, Natural Questions) after pretraining, while matching reasoning performance. This makes MoRE a practical way to scale expert count without sacrificing efficiency.