(MTP) heads are normally trained jointly with the backbone over trillions of tokens, locking drafter quality to pretraining time. This paper shows that a post-training pass on roughly 2.5 billion tokens of the frozen model's own chain-of-thought reaches the same expected speedup as the jointly-pretrained MiMo-7B MTP baseline on math, coding and knowledge benchmarks, using 10^3 to 10^4 times fewer training tokens.
Post-training multi-token prediction heads match joint pretraining with far less data
Lightweight post-training on backbone-generated chain-of-thought data enables frozen reasoning models to achieve comparable throughput speedups.
Big Tech
Prachi Badarayani · Aidan Jay · Chenghui Zhou · Dayquan Julienne · Yuan Gao · Tianwei Chen · +5 more
Microsoft
Research Digest··2 min read
In one line
Post-trained MTP heads on a frozen model match jointly-pretrained quality with 10^3-10^4x fewer tokens; relaxed verification and adaptive head count add speedups.
Before you read this
Three ideas this paper builds onmulti-token predictionTraining a language model to predict multiple future tokens at once, rather than one at a time, improving generation speed and efficiency.6 papers →speculative decodingA method that uses a lightweight model to propose token sequences that are then verified in parallel by a larger model.67 papers →verificationChecking outputs for correctness before accepting them, often to catch errors or reward hacking.345 papers →
Read it as
The authors show that post-training multi-token prediction (MTP) heads on approximately 2.5 billion tokens of target-generated chain-of-thought data matches or exceeds the expected speedup of jointly pretrained MTP heads on math, coding, and knowledge benchmarks, using 10^3 to 10^4 times fewer training tokens.
The authors attached three chained MTP heads to a frozen Qwen3-8B backbone language model and trained the heads using plain cross-entropy loss on approximately 2.5 billion tokens of backbone-generated chain-of-thought data.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.