Speculative decoding accelerates inference by having a small model propose several tokens, which a larger target model verifies together.
History-free drafting improves speculative decoding for long-running AI agents
DLite reuses target-model hidden states to propose tokens without maintaining a separate draft-model history.
Top University
Jiaming Xu · Hanzhen Wang · Tianlang Zhao · Jiewen Xiao · Chengze Yuan · Jiayi Pan · +4 more
Shanghai Jiao Tong University · Shanghai Innovation Institute · Tsinghua University · Infinigence-AI
Research Digest··3 min read
Xu and colleagues developed DLite, a speculative decoding framework designed for agentic workloads with long, repeatedly extended contexts and frequent model calls.
§