Large language models can be slow because each new word requires a full model run. speeds things up by having a small, fast model propose several words at once, which the large model then checks in one go. The speed gain depends on how many proposed words are accepted in a row, called acceptance length.
Existing training methods teach the small model to predict the next word correctly, but they do not directly aim to maximize acceptance length. D-REAL is a new post-training technique that directly optimizes for longer acceptance length. It does this by assigning more importance to word predictions that are more likely to survive the full check, using a mathematical trick to get an unbiased estimate of how to improve.
In tests across ten different question sets, one round of D-REAL training increased average acceptance length by up to 0.23 tokens beyond already-optimized models. This translated into up to 9 percent faster overall system throughput, meaning users get answers faster without changing the hardware or the large model.