Large language models usually generate one token at a time, which can be slow. speeds this up: a smaller draft model suggests tokens, and the larger target model checks them together. Drafts that use the target’s internal representations often copy its hidden width, the size of the vectors passed between their layers.
The team trained two kinds of draft model with narrower internal vectors, then measured generation speed and how many suggested tokens the target accepted. For one design, they combined a conversion layer with the target’s output layer, preserving its scores apart from tiny rounding differences.
Across tested setups, narrower drafts made end-to-end decoding about six to forty-four percent faster, with acceptance dropping by at most about five percent. Draft models need not inherit the target’s size: reducing their internal computation can speed generation, especially on hardware where moving data is a bottleneck.
