Large language models reading very long text must keep a huge history, called a key-value cache. Reading the entire cache is costly, so models pick only the most relevant parts to read. The problem is that this picking itself is a bottleneck: it must wait for the model's current thought to finish before it can start.
EarlySelectKV solves this by using a rough draft of that thought from an earlier model layer to begin the selection process ahead of time. The real thought is still used for the final reading, so accuracy is preserved. This works without retraining and can be dropped into existing systems.
The technique matches the original system's quality almost perfectly. In a real deployment, it improved the speed of generating output by up to a twelfth and freed up over a ninth of the GPU memory previously needed for the history cache. This makes serving long-context models significantly cheaper and faster without any quality trade-off.