PQ-HSA targets the accuracy loss caused by methods that assign zero weight to unselected tokens. It scans an IVF-PQ index to estimate logits for all cached keys, uses exact keys and values for selected tokens, and represents the remaining tokens through per-list approximate weights and mean values under the same softmax.
At 128K context with a 1 to 2% retrieval budget, it outperformed Quest and SnapKV on Llama-3.1-8B and Qwen3-30B-A3B while remaining close to full-attention macro accuracy. On the 8B model, adding the background representation raised macro accuracy from 0.71 to 0.83. In vLLM on one NVIDIA H20, its decode attention call was 1.6 times faster than FlashAttention-3 at 128K, with larger gains at longer contexts.