The paper introduces two methods to improve routing in sparse mixture-of-experts (MoE) . Standard routers learn affinity scores indirectly from the language-model objective, which does not directly account for token-level prediction error. The first method, token-error supervision (TES), trains an auxiliary head to predict per-expert error, weighting these predictions by affinities to align with the cross-entropy loss. The second, affinity-concentration supervision (ACS), directly adjusts the router's affinities using the realized token loss without an additional head. Both use the Itakura-Saito divergence or exponential negative log-likelihood for alignment.
Evaluated on Granite and OLMoE models across four multiple-choice QA benchmarks, both methods improve accuracy. On Granite, ACS-IS improved average accuracy by 2.1 percentage points, while TES-IS improved by 2.3 points over a parameter-matched baseline. The gains are notable (e.g., 2.94 points on ARC-Challenge with stronger supervision) while preserving the sparse execution budget. The results show direct token-loss supervision can complement existing routing, with the best method depending on the backbone and task.