Tree-based speculative decoding accelerates large language model inference by verifying multiple candidate branches in parallel.
ValuSpec filters low-quality candidate branches before verification to speed speculative decoding
The plug-and-play module uses transfer-based supervision to align filtering with the target model without training.
Chinese Tech
Liang He · SiYuan Ma · Qishi Zhan · Yongqi Fan · Mingyu Cao · Qizhen Lan · +2 more
Tongji University · Nanyang Technological University · Marquette University · Tencent · University of Surrey
Research Digest··2 min read
The authors propose ValuSpec, a candidate valuation mechanism that filters low-quality branches from a candidate tree before target-side verification.
§