Mixture-of-Experts (MoE) models use a router to assign each token to a small set of expert parameters. Standard routers compute logits only from the current token representation, ignoring how the same token was routed in earlier layers. HERO-MoE adds a residual contribution from the detached, dense routing distributions of preceding layers before the softmax and top‑k dispatch, with a scale‑preserving fusion mechanism to keep the historical signal stable.
Tested on an 8B‑parameter MoE model with 0.5B active parameters trained from scratch on 100B tokens, HERO-MoE reduced the final validation loss from 1.6393 to 1.6184. Peak memory and FLOPs increased by only 0.44% and 0.64% respectively, and no auxiliary loss or hyper‑parameter tuning was required.
The work is interesting because it shows that inexpensive, already‑computed routing history can provide a clear quality gain for MoE training without altering the standard sparse dispatch procedure or requiring significant architectural changes.