← All concepts

speculative decoding

To speed up text generation, a small, quick model drafts several future words at once. A large, accurate model then verifies the entire batch. Because checking a block of guesses is much faster than writing it word by word, the whole process runs faster.

The papers tackle the bottlenecks that limit this basic idea. They find that the drafter itself must be carefully optimized to avoid wasting effort, especially on long inputs or changing conditions. Strategies include keeping the drafter's cost constant regardless of context, letting it adapt its drafting strategy on the fly, or giving it special access to images and prior computation. The common thread is shifting focus from raw acceptance length toward reducing overhead and wasted computation.

Papers
62
Released code
0
First seen
Aug 2026
Latest
Nov 2026

60 papers in the last two months, against 2 in the two before.

Start here

Three papers to read first, in the order they came out
  1. 1Language models generate 5–8% faster by keeping draft alternatives separate

    Multi-Future Drafting avoids chimeras in speculative decoding by using mode embeddings, delivering up to 7.8× speedup.

    Nov 2026
  2. 2Language models can nearly double output speed with fewer cloud queries

    SpecRelay amortizes or avoids costly round trips in edge-cloud speculative decoding to boost throughput under variable network latency.

    Nov 2026
  3. 3Speculative decoding speeds up vision models by 3.58 times with visual-token weighting

    MMFlash speeds up vision-language model inference by up to 3.58x using adaptive weight adjustments for visually dependent tokens.

    Nov 2026

Who is working on it

NVIDIA 4University of Science and Technology of China 3Google 3Shanghai Innovation Institute 2Renmin University of China 2Huawei Technologies Co., Ltd. 2

The papers

Most central to this idea first, not most recent.

Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.