The paper first presents a theoretical analysis of when a target probability row (from a verification pass) can be reused exactly for the next round's tree selection.
Reusing target probabilities in speculative decoding can improve tree selection but requires careful trade-offs.
A study finds that reranking draft branches using target model probabilities from the previous pass increases accepted tokens, especially with small tree budgets, but exact reuse requires sacrificing accepted tokens.
Academic
Jungseob Lee · Giyoon Jang · Chailin Nam · Jiwoo Kum · Sihyeon Lee · Kyojun Choo · +3 more
Korea University · Soongsil University
Research Digest··3 min read
The authors investigate when next-token probabilities from a target model's verification pass can be reused to select the next tree of draft tokens.
§