Long-context language models generate text one word at a time, reading a large memory of recent tokens. To speed this up, some methods only read a subset of that memory, but they have no way to check if the subset captured enough information, leading to errors that compound. A new approach uses a number the model already computes, the fraction of attention weight on the selected tokens, called coverage, to verify the output. If coverage is high, the output is accepted; if low, only the failing parts are recalculated. This method, called Coverage-Verified , matches the quality of full memory within 1.13 points on benchmark tests while achieving a median decode-step speedup of 6.21 times at 128K context, without needing extra models or training.
Sparse attention for long-context AI verifies its own quality in one pass
A training-free method that checks and corrects sparse attention outputs per step, matching dense attention within one point on long-document benchmarks while decoding up to six times faster
Academic
Yan-Cheng Li · Chun-Yi Lee
National Taiwan University
Research Digest··3 min read
In one line
Coverage-Verified Sparse Attention achieves closed-loop quality control by using the softmax mass fraction to verify and correct sparse attention outputs.
Before you read this
Three ideas this paper builds onsparse attentionSelectively attending to only the most relevant positions or tokens to reduce computational overhead.50 papers →verificationChecking outputs for correctness before accepting them, often to catch errors or reward hacking.323 papers →kv cache compressionReducing the size of the key-value cache in transformers via selective retention, low-rank updates, or adaptive compression.78 papers →
Read it as
Li and Lee present Coverage-Verified Sparse Attention (CVSA), a closed-loop decode method that uses a scalar already computed inside the sparse-attention kernel, the fraction of softmax mass on selected tokens, as a quality signal.
The authors observed that kernels already compute the fraction of softmax mass on selected tokens, which they call coverage, but discard it.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.