The authors start from a looped MoE model (based on SMELT) that applies the middle half of its layers twice with shared parameters.
Asymmetric compute allocation improves looped Mixture-of-Experts models
LatentLoop compresses the first visit's sequence, retains full resolution in the second, and reallocates saved computation to increase hidden size, consistently lowering validation NLL across 15 configurations.
Chinese Tech
Zefeng He · Yunkai Chai · Xiaoyang Chen · Tong Zhu · Zongliang Wu · Yun Luo · +2 more
SJTU · Shanghai AI Lab · USTC · CUHK · NTU
Research Digest··3 min read
The authors probe the contributions of different loop visits in a looped Mixture-of-Experts (MoE) transformer, finding that the first visit benefits from a reduced residual scale while the second does not.
Why this paper
From Shanghai AI Lab and 4 others
In one line
Compressing the first loop visit and spending the saved compute on hidden width lowers validation loss in looped mixture-of-experts models across budget sizes.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§