A Vestigial Attention Branch Signals Which Cache Entries to Keep

What they did

VestigeKV ranks cache rows using Kimi Linear’s 64-dimensional decoupled attention branch, reading about 11% of each row. The authors argue that, after training without positional encoding (NoPE), this vestigial RoPE-related branch becomes a query-independent token-salience signal.

The highest-ranked rows remain in an attended tier; all others move bit-exactly to a GPU-resident archive that can be consulted when a trigger fires. The method requires no retraining, quantization, weight changes, or new kernels. It was tested on needle retrieval at context lengths from 8k to 65k, with attention-based eviction methods and a RoPE MLA model used as comparisons.

Key findings

Retrieval remained 1.00 at 8× attended-tier compression and 0.92 at 32×, with no reported gap between selection using the 64-dimensional branch and full cache rows.
A recall-tier configuration achieved 1.00 retrieval at 128× compression.
At 32×, the attended tier used 0.25 KB per token, compared with Kimi Linear’s full 8.1 KB per-token cache; archived rows remained exact and GPU-resident unless explicitly offloaded to host memory.
Prior attention-based selectors scored 0.00–0.33 on the NoPE setup. Applying VestigeKV’s operator to a RoPE MLA model scored 0.08, versus 0.42 for plain eviction, indicating that the signal is specific to the unrotated architecture.

Why it matters

Long-lived caches may need to be compressed before the queries revealing each token’s importance exist, limiting methods based on past attention. VestigeKV suggests that some NoPE attention architectures already encode a usable, query-independent salience signal, allowing a small active cache while retaining exact access to less likely entries.

Caveats

The evidence is limited to the measured Kimi Linear model and synthetic needle retrieval; the paper does not establish equivalent gains on broad language tasks or other NoPE architectures. Because the default archive remains on the GPU, attended-tier compression does not by itself reclaim all VRAM; host offloading is required for that. Extension to Kimi K3 is explicitly speculative, and the method performs poorly when transferred to RoPE MLA.