multi-token prediction
For an AI that predicts one word at a time, multi-token prediction means guessing several future words in one go. Normally, the model learns to do this during its initial enormous training run. The idea is that predicting a chunk of text at once forces the model to plan ahead, which can make it better at tasks like reasoning, and also speeds up text generation because the model can accept or reject its guesses in bulk.
The papers show two main trends. First, the prediction heads used for guessing future tokens can be taught after the main model is trained, using far less data than previously needed. Second, the same model that does the main prediction can also serve as its own guesser for faster generation, either through clever architecture changes that reuse the main network, or by splitting the workload between a phone and a server. All approaches boost generation speed and often improve reasoning quality.
- Papers
- 6
- Released code
- 0
- First seen
- Oct 2026
- Latest
- Nov 2026
6 papers in the last two months, against 0 in the two before.
Start here
Three papers to read first, in the order they came out- 1Shared language-model layers make multi-token prediction cheaper and more direct
ECHO reuses the backbone's final layer for multi-token prediction and uses dynamic depth sampling, improving performance and enabling efficient speculative decoding.
Oct 2026 - 2One-line fix makes language models learn better and decode faster for free
Soft-MTP improves multi-token prediction training by filtering out noisy gradient signals, boosting performance and inference speed without added cost.
Oct 2026 - 3Language models can generate text up to three times faster
Multi-token prediction via a shared drafting head achieves up to 3.1x speedup for LLM inference on servers and up to 2.2x on smartphone GPUs.
Nov 2026
How fast it is moving
Often studied with
speculative decodingA method that uses a lightweight model to propose token sequences that are then verified in parallel by a larger model.6 togetheron-policy distillationA training method where a student model learns from a teacher using its own current policy's outputs, often improving reasoning and instruction following.1 togetherquantization-aware trainingTraining a neural network with simulated quantization in the forward pass so its weights and activations adapt to low-precision formats, improving deployment accuracy.1 togetherThe papers
Most central to this idea first, not most recent.
- Big Techcs.CL
Language models can generate text up to three times faster
Multi-token prediction via a shared drafting head achieves up to 3.1x speedup for LLM inference on servers and up to 2.2x on smartphone GPUs.
Google · Nov 2026
- Independentcs.LG
One-line fix makes language models learn better and decode faster for free
Soft-MTP improves multi-token prediction training by filtering out noisy gradient signals, boosting performance and inference speed without added cost.
Oct 2026
- Chinese Techcs.CL
Shared language-model layers make multi-token prediction cheaper and more direct
ECHO reuses the backbone's final layer for multi-token prediction and uses dynamic depth sampling, improving performance and enabling efficient speculative decoding.
ERNIE Team, Baidu, Independent Researcher · Oct 2026
- Big Techcs.LG
Post-training multi-token prediction heads match joint pretraining with far less data
Post-trained MTP heads on a frozen model match jointly-pretrained quality with 10^3-10^4x fewer tokens; relaxed verification and adaptive head count add speedups.
Microsoft · Oct 2026
- Big Techcs.CL
Hybrid drafters split depth and breadth to cut mobile-cloud GPU costs
HyCo reduces server GPU time by 38.6% using a hybrid drafter with MTP head and SLM on mobile.
Samsung Research · Oct 2026
- Independentcs.CL
Scalable flow language model rivals autoregressive quality at 2.5x speed
Continuous flow language models can scale to 8B parameters, match autoregressive quality, and decode in parallel faster.
Oct 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.