The authors introduce Mixture of Rank-reduced-routed Experts (MoRE), which factorizes the router weight matrix at rank r.
Low-rank router enables scaling mixture of experts to many more experts
The authors show that factorizing the router matrix at logarithmic rank reduces routing cost without sacrificing expressivity or load balance.
Top University
Honam Wong · Surbhi Goel · Enric Boix-Adserà
University of Pennsylvania
Research Digest··3 min read
The authors propose MoRE, which replaces the standard linear router in mixture-of-experts layers with a low-rank factorization, reducing per-token cost from Θ(Mh) to O((h+M)r).
Why this paper
From University of Pennsylvania
In one line
Low-rank routing in mixture-of-experts layers enables scaling to many more experts without increasing active compute.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§