The authors work with LatentMoE, where experts operate in a lower-dimensional space, decoupling input dimension and intermediate width.
Near-square experts and depth-aware allocation improve MoE models
Balancing expert input and intermediate dimensions maximizes representational capacity, and layerwise intrinsic dimension guides efficient capacity allocation.
Chinese Tech
Yaoxiang Wang · Kai Zhang · Ruobing Xie · Lanrui Wang · Guoliang Zhao · Junhui Li · +1 more
Xiamen University · Tencent Hunyuan · Soochow University
Research Digest··3 min read
The authors study expert shape in LatentMixture-of-Experts (MoE) models, showing that when a fixed parameter budget is split between an expert's input dimension and intermediate width, a near-square balance maximizes the rank of the expert's Jacobian.
Why this paper
From Tencent Hunyuan and 2 others
In one line
For latent mixture-of-experts, near square experts maximize per parameter rank, and layerwise intrinsic dimension guides capacity allocation.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (2 noted)
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§