Language models can accelerate more when handling one request alone

Across 10 sparse mixture-of-experts models, the authors find that many achieve their greatest speculative-decoding speedup at or near a batch size of one, contrary to earlier predictions.

AI Startup

Ekagra Ranjan · Komal K. Teru · Bharat Venkitesh · Acyr Locatelli

Cohere

Research Digest·30 October 2026·3 min read

Illustration: Language models can accelerate more when handling one request alone

In one line

Speculative decoding on sparse MoE models remains fast at low batch size because routing overlap and GPU execution effects make verification cheaper than activated-expert bytes suggest.

Before you read this

Two ideas this paper builds on

speculative decodingA method that uses a lightweight model to propose token sequences that are then verified in parallel by a larger model.39 papers →mixture of expertsA neural architecture with many specialized sub-networks, only a few activated per input via a router, balancing capacity and efficiency.63 papers →

Read it as

Language models generate text one token at a time. uses a small draft model to propose several tokens; the big model verifies them in one pass. In mixture-of-experts (MoE) models, each token uses a few specialized experts. Old theory predicted little benefit at small batch sizes, where few requests are handled together.

Tests on ten sparse MoE models, thirty billion to one trillion parameters, found recent fine-grained models stay fast at batch size one. Adjacent tokens share experts far more often than chance, so far fewer unique experts need loading. The GPU cache absorbs repeated reads.

Fine-grained MoE designs suit low-batch speculative inference. A statistical model predicts unique expert counts within 4% error, allowing design screening without training.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

speculative decoding mixture of experts