speeds up language model inference by using a fast drafter model, but traditionally must match the target's output distribution. Mentored decoding relaxes this constraint, allowing a controlled drift to further improve speed, and in practice has been observed to also improve quality. This paper provides the first formal proof that such quality improvement is possible, by connecting the problem to theory from machine learning.
The authors generalize mentored decoding to any f-divergence, and show that the optimal output distribution takes a simple, coordinate-wise clamping form that depends only on the drafter and target distributions, not on the specific divergence. They also present an efficient data structure, requiring O(n) space and O(sort(n)) time to build, that can query optimal parameters in O(log n) time and construct optimal mentored distributions in O(n) time for any f-divergence.
The key insight is that even a small divergence from the target can yield a significant boost in acceptance probability, explaining how the composite model can not only run faster but also produce higher quality outputs. This formal link between inference and boosting opens new avenues for model combination in LLM serving.