Large language models generate text token by token, which is slow. speeds this up by having a small, fast 'drafter' model propose several tokens at once, then a larger 'target' model verifies them to ensure quality. This paper focuses on hybrid target models that combine two types of layers: Gated DeltaNet (GDN), which maintains a compact memory of past tokens, and mixture-of-experts (MoE), where each token only uses a small fraction of the model's total parameters. The critical choice in each round is how many draft tokens to check at once.
The authors introduce HybridSpec, which picks this number on the fly based on the drafter's confidence and the measured time the target takes. It first runs verification on a slightly longer block, then after the first MoE layer, it can discard a token if the cost of checking it further exceeds its expected benefit. They also created new GPU kernels that can verify all tokens in a round simultaneously for the GDN layers, avoiding the usual step-by-step memory updates.
For three GDN-MoE models tested, HybridSpec generated tokens about 1.75 to 1.88 times faster than the standard one-by-one method, and 1.30 to 1.33 times faster than a baseline speculative decoding approach. This means much faster text generation without changing the model's output quality, making these complex models more practical for real-time applications.