speculative decoding
To speed up text generation, a small, quick model drafts several future words at once. A large, accurate model then verifies the entire batch. Because checking a block of guesses is much faster than writing it word by word, the whole process runs faster.
The papers tackle the bottlenecks that limit this basic idea. They find that the drafter itself must be carefully optimized to avoid wasting effort, especially on long inputs or changing conditions. Strategies include keeping the drafter's cost constant regardless of context, letting it adapt its drafting strategy on the fly, or giving it special access to images and prior computation. The common thread is shifting focus from raw acceptance length toward reducing overhead and wasted computation.
- Papers
- 62
- Released code
- 0
- First seen
- Aug 2026
- Latest
- Nov 2026
60 papers in the last two months, against 2 in the two before.
Start here
Three papers to read first, in the order they came out- 1Language models generate 5–8% faster by keeping draft alternatives separate
Multi-Future Drafting avoids chimeras in speculative decoding by using mode embeddings, delivering up to 7.8× speedup.
Nov 2026 - 2Language models can nearly double output speed with fewer cloud queries
SpecRelay amortizes or avoids costly round trips in edge-cloud speculative decoding to boost throughput under variable network latency.
Nov 2026 - 3Speculative decoding speeds up vision models by 3.58 times with visual-token weighting
MMFlash speeds up vision-language model inference by up to 3.58x using adaptive weight adjustments for visually dependent tokens.
Nov 2026
How fast it is moving
Often studied with
verificationChecking outputs for correctness before accepting them, often to catch errors or reward hacking.13 togetherkv cache compressionReducing the size of the key-value cache in transformers via selective retention, low-rank updates, or adaptive compression.8 togetherdiffusion modelA type of generative model that learns to reverse a gradual noising process to create data.7 togetherWho is working on it
The papers
Most central to this idea first, not most recent.
- Big Techcs.AI
Language models can nearly double output speed with fewer cloud queries
SpecRelay amortizes or avoids costly round trips in edge-cloud speculative decoding to boost throughput under variable network latency.
Georgia Institute of Technology, Apple · Nov 2026
- Top Universitycs.CV
Speculative decoding speeds up vision models by 3.58 times with visual-token weighting
MMFlash speeds up vision-language model inference by up to 3.58x using adaptive weight adjustments for visually dependent tokens.
National University of Singapore · Nov 2026
- AI Startupcs.AI
Language models generate 5–8% faster by keeping draft alternatives separate
Multi-Future Drafting avoids chimeras in speculative decoding by using mode embeddings, delivering up to 7.8× speedup.
Beihang University, Canva Research · Nov 2026
- Academiccs.AI
Flow matching speeds up speculative decoding by 39% over existing drafters
Drafting speculative tokens as a short flow over the target model's hidden states speeds generation faster than autoregressive or diffusion drafters.
Yildiz Technical University · Nov 2026
- Independentcs.DC
Stale model memory as draft cuts AI agent delays after context edits
Callo reduces cache-miss latency by speculating from invalidated KV cache while prefill computes the exact cache.
Nov 2026
- Chinese Techcs.AI
Language models can generate text five times faster by drafting ahead
D3D uses a decoupled diffusion drafter for asynchronous speculative decoding, achieving average speedups of 5.00x and 5.71x.
Shanghai Innovation Institute, Shanghai Academy of AI for Science · Nov 2026
- Independentcs.AI
Directly optimizing acceptance length boosts speculative decoding throughput by up to 9.4%
D-REAL post-training directly optimizes expected accepted draft token length to speed up speculative decoding.
Nov 2026
- Independentcs.RO
Flash decoding speeds visual planning for robots 25x
VFlash speeds up visual planning by predicting entire spatial primitives in one forward pass rather than one token at a time.
Nov 2026
- Chinese Techcs.AI
Parallel training method speeds language model inference by 3.5x
SpecOPD decouples on-policy supervision from sequential rollouts, enabling parallel distillation for speculative decoding.
Alibaba, University of Science and Technology of China · Nov 2026
- Big Techcs.LG
Language models run faster when small helpers revisit their uncertain guesses
Parallel Reconsideration improves speculative decoding by letting the drafter revisit uncertain tokens with extra parallel passes, increasing acceptance length and speedup.
Rice University, Google · Nov 2026
- Chinese Techcs.CL
AI agents’ average agreement can mask declines in later browsing actions
TraceSpec shows that allocating draft supervision by trajectory state and prefix-aware acceptance gaps raises agent speculative-decoding throughput by 12.5% to 22.6% over DSpark.
Renmin University of China, Huawei Technologies Co., Ltd. · Nov 2026
- AI Startupcs.LG
Language models can accelerate more when handling one request alone
Speculative decoding on sparse MoE models remains fast at low batch size because routing overlap and GPU execution effects make verification cheaper than activated-expert bytes suggest.
Cohere · Oct 2026
- Big Techcs.AI
Fewer draft tokens verified per round makes hybrid models a third faster
HybridSpec speeds up hybrid GDN-MoE language model inference by adaptively selecting verification length and using parallel GDN kernels.
Meta AI, New York University · Nov 2026
- Chinese Techcs.CL
Speculative decoding throughput gains are conditional on workload and serving conditions
In speculative decoding, extra acceptance pays only when its throughput gain exceeds added latency; shared training and inference conditions shrink method gaps.
Tianjin University of Technology, Baidu Inc. · Oct 2026
- Big Techcs.CL
On-device AI drafters can accept more tokens with less memory by separating context from local attention
HyFlash achieves higher acceptance than full-context attention with less KV storage, especially for shallow drafters and long contexts.
Microsoft Applied Sciences Group · Nov 2026
- Academiccs.CL
Reusing target probabilities in speculative decoding can improve tree selection but requires careful trade-offs.
Target probabilities improve speculative-decoding tree selection, but guaranteed exact reuse requires a distribution computed at the next round’s full context.
Korea University, Soongsil University · Oct 2026
- Big Techcs.LG
Jointly co-compiling draft and target models boosts speculative decoding speed
Conjugate Compilation jointly optimizes draft and target models for speculative decoding, achieving up to 1.59x speedup on single GPUs.
University of California, San Diego, Massachusetts Institute of Technology · Oct 2026
- Chinese Techcs.CL
Progressive verification speeds up diffusion language model inference
PV-Decoding reduces sequential model forward passes by 8-37% while keeping the largest observed accuracy drop to 1.3% for diffusion language models.
Huawei Technologies, City University of Hong Kong · Oct 2026
- Chinese Techcs.AI
Language models can give identical answers up to 2.78 times faster
Filtering low-quality branches from a candidate tree before expensive verification speeds up tree-based speculative decoding without changing outputs.
Tongji University, Nanyang Technological University · Oct 2026
- Big Techcs.CL
Hybrid drafters split depth and breadth to cut mobile-cloud GPU costs
HyCo reduces server GPU time by 38.6% using a hybrid drafter with MTP head and SLM on mobile.
Samsung Research · Oct 2026
- Top Universitycs.CL
History-free drafting improves speculative decoding for long-running AI agents
DLite improves speculative decoding for agentic workloads by reusing target-model hidden states instead of maintaining a draft model’s historical KV cache.
Shanghai Jiao Tong University, Shanghai Innovation Institute · Oct 2026
- Big Techcs.CL
Refreshing compressed draft caches preserves alignment during long language model outputs
RIFT compresses draft-model KV caches by mixing prefill and recent tokens, refreshing selection, and training with sparse masks to preserve acceptance length.
NVIDIA · Oct 2026
- Chinese Techcs.CL
Tree-trained draft branches make speculative decoding accept longer continuations
TreeSpark improves speculative decoding throughput by 15-25% over DSpark by training drafters on model-generated parent states with tree-structured candidates.
Renmin University of China, National Key Lab for Novel Software Technology, Nanjing University · Oct 2026
- Independentcs.LG
Fixed-cost drafter speeds speculative decoding as context lengths grow
LongSpark decouples drafter cost from prefix length by extracting fixed-size, multiscale views from target verification.
Oct 2026
- Big Techcs.CL
Learned controller adapts speculative decoding to maximize speed and reduce waste
APEX adaptively selects speculation mechanism and draft depth to balance speed and token waste.
University of Illinois Urbana-Champaign, AWS AI · Oct 2026
- Big Techcs.CL
Iterative draft editing speeds up speculative language model decoding
DEdit improves speculative decoding speed by iteratively editing drafts with bidirectional context, achieving over 5.7x speedup on Qwen3 models.
University of California, Riverside, Amazon Web Services · Oct 2026
- Chinese Techcs.LG
Language models improve at math when teachers steer their practice
Coupling student and teacher rollouts with a rejection-sampling rule raises math benchmark accuracy above on-policy distillation.
Tsinghua University, Harbin Institute of Technology, Shenzhen, China · Nov 2026
- Big Techcs.LG
Post-training multi-token prediction heads match joint pretraining with far less data
Post-trained MTP heads on a frozen model match jointly-pretrained quality with 10^3-10^4x fewer tokens; relaxed verification and adaptive head count add speedups.
Microsoft · Oct 2026
- Research Labcs.DC
Adaptive verification speeds block-diffusion decoding under heavy concurrent demand
DScale improves throughput of block-diffusion speculative decoding by up to 48.8% over DFlash with only a 112K-parameter predictor.
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, University of Chinese Academy of Sciences · Oct 2026
- Chinese Techcs.LG
Depth-wise predecessor conditioning makes parallel speculative drafts more consistent
DSpine injects causal conditioning at every layer, increasing acceptance length by 27.8% and throughput by 23.3% over DFlash.
Shanghai Jiao Tong University, Tencent YouTu Lab · Oct 2026
- Chinese Techcs.CL
Looped diffusion drafting improves decoding without extra model components
D-Loop adds intra-block causal conditioning to diffusion drafting, improving acceptance length and decoding speed without extra model components.
City University of Hong Kong, The Chinese University of Hong Kong · Oct 2026
- Big Techcs.CV
HAWK boosts speculative decoding for LVLMs via target hidden states and prediction-shift training
HAWK improves multimodal speculative decoding by feeding drafters compressed target visual hidden states and supervising prediction shifts, raising acceptance length and speedup.
University of California, Los Angeles, Samsung Research America · Oct 2026
- Independentcs.AR
Fixed chip beats general GPU by 690x on language model tokens
Exstill decodes tokens 110x faster at 690x lower energy than a B300 by co-designing model and circuits.
Nov 2026
- Top Universitycs.DC
MoE model disaggregation yields modest speedup on rack-scale NVLink, model predicts payoff
On rack-scale NVLink, attention-FFN disaggregation can deliver up to 1.17x more throughput than wide expert parallelism for some MoE models.
University of California, Berkeley · Nov 2026
- Academiccs.CL
Draft-training objective gains shrink when target regenerates training responses
Draft-training objectives' gains in speculative decoding largely disappear when training on target-regenerated responses.
Korea University, Yonsei University Mirae Campus · Oct 2026
- Big Techcs.CL
AI coding model beats predecessor using roughly one-fifth the active parameters
Nemotron 3.5 Super trains a 12B-active multimodal model that exceeds a 4.6x larger model on SWE-bench Pro.
NVIDIA · Oct 2026
- Independentcs.LG
One-line fix makes language models learn better and decode faster for free
Soft-MTP improves multi-token prediction training by filtering out noisy gradient signals, boosting performance and inference speed without added cost.
Oct 2026
- Industrycs.CR
New watermark for language models detects AI text without distorting output or slowing decoding
TextSeal2 merges multiple watermark keys into a single virtual key, matching state of the art detection without per token optimization or lowered speculative decoding acceptance.
Meta Superintelligence Labs · Oct 2026
- Chinese Techcs.CL
Shared language-model layers make multi-token prediction cheaper and more direct
ECHO reuses the backbone's final layer for multi-token prediction and uses dynamic depth sampling, improving performance and enabling efficient speculative decoding.
ERNIE Team, Baidu, Independent Researcher · Oct 2026
- Big Techcs.LG
Replaying cached inputs cuts state memory traffic in SSM decoding
Representing an SSM state as a checkpoint plus a recent-input buffer cuts state memory traffic, enables cheap speculative rollback, and keeps state quantization error out of later updates.
Princeton University, NVIDIA · Oct 2026
- Big Techcs.LG
A learned router chooses when language models should draft farther ahead
CADENCE learns context-adaptive decode length by generating Gaussian latent spans, achieving up to 1.3x higher generation throughput than greedy decoding.
Microsoft, University of Massachusetts Amherst · Oct 2026
- Top Universitycs.LG
Speculative parallelism boosts multi-GPU LLM inference efficiency
Spexis accelerates multi-GPU LLM serving by overlapping speculative and normal execution and scheduling around predicted speculation quality and memory pressure, reaching 34% speedups.
Seoul National University, Hanyang University · Oct 2026
- Chinese Techcs.CR
Simulated future actions may expose multi-turn agent attacks earlier
SSH simulates future agent trajectories and verifies them against real actions to detect multi-turn attacks earlier while reducing false positives.
Huawei Technologies Co., Ltd · Oct 2026
- Chinese Techcs.AI
Full-context drafters help compressed agent models retain accuracy
AsymSpec uses an asymmetric drafter and verifier to recover full-context accuracy at compressed inference cost.
Huawei Technologies Co., Ltd., University of Science and Technology of China · Aug 2026
- Academiccs.LG
New method makes quantized AI models up to 31% smaller with no quality loss
LAZYWEIGHTS tunes GPTQ quantized weights for Huffman compression, reducing edge LLM memory footprint by 5.7–31.4% at equal perplexity.
University of Texas at Austin, Case Western Reserve University · Nov 2026
- Independentcs.LG
Language models can start answering up to 1.6 times faster
RidgeFill speeds prefill by predicting late-layer inputs from early-layer states with a regression fit once from calibration text.
Independent researcher · Nov 2026
- Chinese Techcs.CR
Private-query AI assistants leak secrets through rejected drafts, new protocol plugs the hole
ADLER ensures every message to the cloud depends only on the public view and the answer the user sees.
Australian Catholic University, Ant Group · Oct 2026
- Independentcs.CL
Scalable flow language model rivals autoregressive quality at 2.5x speed
Continuous flow language models can scale to 8B parameters, match autoregressive quality, and decode in parallel faster.
Oct 2026
- Chinese Techcs.CV
Combining compression and adaptive inference for diffusion transformers without quality loss
Trajectory-aware student training with learned routing and teacher rescue makes compressed Diffusion Transformers compatible with adaptive execution, reaching FID 2.58 at 2.25× speedup.
The Hong Kong University of Science and Technology (Guangzhou), University of Electronic Science and Technology of China · Oct 2026
- Independentcs.LG
Temporal indexer prediction speeds up long-context sparse attention by 14.9%
SPIN predicts which KV cache blocks are important using past scores, reducing indexer overhead by 30-40% and improving throughput by up to 14.9%.
Oct 2026
- Top Universitycs.CL
Byte Transformers outperform subword models at scale and learn implicit token-like abstractions
Byte Transformers with token-superposition training and hash embeddings outperform subword Transformers while building internal text abstractions.
East China Normal University, Fudan University · Oct 2026
- Big Techcs.LG
Mentored decoding can trade small distribution shifts for better predictions
Mentored decoding proves that lossy speculative decoding can beat the target model in quality while speeding inference.
Google · Oct 2026
- Academiccs.LG
Sparse attention for long-context AI verifies its own quality in one pass
Coverage-Verified Sparse Attention achieves closed-loop quality control by using the softmax mass fraction to verify and correct sparse attention outputs.
National Taiwan University · Nov 2026
- Top Universitycs.LG
New system recovers wasted GPU time in RL training of language models
TIDE recovers GPU utilization lost to straggler rollouts and inefficient decoding in on-policy LLM reinforcement learning training.
Stanford University, University of California, Berkeley · Nov 2026
- Big Techcs.LG
Language models can train up to 24% faster on Google chips
Systolic Linear Attention runs 1.6x to 2.5x faster on TPUs than existing gated delta attention by widening keys and grouping decay gates.
Google · Nov 2026
- Top Universitycs.LG
Parallel language models match autoregressive quality while decoding several tokens per pass
Parallel Language Models generate coordinated tokens per forward pass at autoregressive-level quality, delivering 2.3-5.7× faster inference in reported post-trained models.
University of California, Irvine, ETH Zurich · Oct 2026
- Independentcs.CV
Parallel diffusion decoding rivals autoregressive visual grounding at 4B scale
GroundAnything uses blockwise diffusion to deliver faster parallel visual grounding while retaining strong localization accuracy across diverse tasks.
Oct 2026
- Chinese Techcs.LG
Blending teacher and student token distributions improves distillation of language models.
Interpolating student and teacher next-token distributions during rollout yields better distillation than using either endpoint alone.
Tencent Inc., University of Science and Technology of China · Oct 2026
- Big Techcs.LG
Semi-autoregressive continuous diffusion models match discrete baselines and enable KV caching.
Continuous diffusion language models with semi-autoregressive position-dependent noise schedules achieve state-of-the-art likelihood and benchmark performance while enabling variable-length generation and KV caching.
NVIDIA, Cornell University · Oct 2026
- Big Techcs.CR
Sampled text can steer black-box models past safety safeguards
Text-only LLM interfaces remain vulnerable to decoding-time jailbreaks when they allow repeated sampling and continuation from supplied assistant prefixes.
University of Southern California, Adobe · Sept 2026
- Independentcs.LG
Discrete diffusion accelerates language models while preserving their autoregressive distribution
Diffusion-augmented LLMs (Uno) achieve up to 3x lossless speedups over autoregressive models by generating multiple tokens in parallel.
Sept 2026
- Top Universitycs.LG
Language models benefit from repeated layers, but more can hurt
Looping transformer layers improves parameter efficiency but requires careful design to avoid quality loss and parallel inference slowdown.
FAIR at Meta, MIT · Oct 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.