Large language models generate text one word at a time, which is slow and wastes computing power. speeds this up by having a smaller helper model guess several future words, which the main model then checks all at once. But existing helpers need to re-process the entire input, adding extra work and memory.
This paper introduces a helper that attaches directly to the main model's final layers, sharing its memory and avoiding the extra processing. The helper uses a shared small transformer, takes the main model's last internal signal as a starting point, and reduces the vocabulary size during guessing by first picking a cluster of likely words then the specific word.
On large server systems the approach speeds up generation by up to 3.1 times, and on a mobile phone GPU by up to 2.2 times. It also includes a method to keep the helper accurate when the main model is fine-tuned for a new task, requiring almost half the extra training passes that tuning the main model alone would need.
