Large language models generate text one token at a time, which is slow. uses a small drafter model to guess multiple future tokens, which the large model then checks in parallel. On memory-constrained devices, the drafter has very limited storage for context.
HyFlash is a compact drafter that separates long-range context from local interactions. It uses a fixed-size memory to remember earlier context and local attention for recent tokens and intra-block relationships. This keeps memory use bounded.
HyFlash achieves higher acceptance of drafted tokens than full-context attention, while using less memory for context. The gains are largest for shallow drafters and long contexts. This enables faster inference on devices without sacrificing quality.