The authors established a comparison protocol covering training, inference, and evaluation.
Speculative decoding throughput gains are conditional on workload and serving conditions
A controlled study across six drafter recipes and ten benchmarks shows that aligning training and inference conditions narrows method gaps, and acceptance improvements only translate to higher throughput under specific task uncertainty and execution contexts.
Chinese Tech
Yongchao Long · Chunrui Zheng · Yuxuan Zhou · Guangkun Nie · Tian-Chi Ma · Chao Pang · +7 more
Tianjin University of Technology · Baidu Inc. · Shandong University · Peking University · Nanjing University
Research Digest··2 min read
The authors systematically evaluate when additional acceptance length in speculative decoding is worth the computational cost.
§