Large language models can use many specialized sub-networks called experts, with a router that chooses a few experts for each word. The router may select poorly because it evaluates experts from only one viewpoint.
To fix this, multiple independent routers each score the experts from different perspectives. Their scores are combined to choose experts, and the pattern of scores also adjusts what each expert receives as input.
The approach improved average performance across nine tasks by 1.18 percentage points over the standard method, without extra computation per word. This yields more accurate outputs without added cost.