kv cache compression

The papers

Most central to this idea first, not most recent.

• Big Techcs.LG

  AI agents can keep needed context yet fail to use it

  Long-horizon agents need both planned token retention and bounded positional distance, because cached context can become unusable when it is too far away.

  Amazon Web Services · Nov 2026

• Independentcs.LG

  Encoder loop adds compute without growing memory in language models

  Looped CEDs achieve recurrent depth with zero round-wise KV growth and match looped Transformer quality.

  Nov 2026

• Academiccs.LG

  Mid-depth layers judge KV-cache evictions better than the owning layer

  Mid-depth layers score KV-cache eviction better than each layer scoring its own cache, improving long-context accuracy at small and mid-sized models.

  Colby College · Nov 2026

• Independentcs.LG

  Language models can start answering up to 1.6 times faster

  RidgeFill speeds prefill by predicting late-layer inputs from early-layer states with a regression fit once from calibration text.

  Independent researcher · Nov 2026

• Chinese Techcs.CL

  Training a gate to drop AI memory defaults to dropping nothing

  A gate for KV cache eviction succeeds only when dense attention is unavailable to it, it learns from dense model predictions, and evicted content re-enters the same softmax.

  The Hong Kong University of Science and Technology, New York University · Nov 2026

• Top Universitycs.LG

  Language models preserve more retrieval accuracy with 97-fold smaller caches

  Extreme KV-cache compression is practical using finite scalar quantization, achieving accurate retrieval and fast decoding.

  The University of Edinburgh · Nov 2026

• Big Techcs.AI

  Affine folding cuts KV cache decoding latency by up to 3.4x on Hopper and Blackwell GPUs

  FOSCAR speeds attention over compact caches by moving shared quantization corrections out of per-value reconstruction while preserving the exact result.

  FutureMLS Lab, Microsoft Research · Nov 2026

• Independentcs.AI

  Native FP4 KV cache keeps long-context LLM accuracy while doubling throughput

  BitDecoding-FP4 enables accurate native FP4 KV cache decoding, achieving up to 3.0x throughput over BF16 KV.

  Nov 2026

• Big Techcs.CV

  Three-axis compression lets medical AI agents run 8x larger batches on a single GPU

  TACT-KV compresses visual KV caches for 3D medical scans by applying a 3D cosine transform and adaptive bit allocation.

  Vanderbilt University, NVIDIA · Nov 2026

• Chinese Techcs.LG

  Language models can cut cache memory nearly 27-fold, retaining most performance

  LowRAM compresses the KV cache by predicting values from keys and using low-rank keys, achieving 26.9x compression with 94.5% baseline performance.

  Huawei, Noah’s Ark Lab, University College London · Nov 2026

• Chinese Techcs.LG

  Coordinating cache precision cuts agents' memory use by a third

  CoMixKV jointly allocates precision across agent KV caches under a shared memory budget, improving accuracy and memory efficiency.

  Xiamen University, Tencent Inc. · Nov 2026

• Big Techcs.CV

  Video generators can nearly halve history storage while improving reconstruction quality

  A compression scheme for cached attention state in autoregressive video models preserves generation quality better by tuning Keys to match downstream outputs rather than to reconstruct the raw cache.

  Zhejiang University, NVIDIA · Nov 2026

• Big Techcs.LG

  Speech language models can drift as audio retrieval mistakes accumulate

  C3 outperforms other KV retrieval methods on long-form speech tasks by clustering audio keys and having heads compete.

  University at Buffalo, Amazon AGI · Nov 2026

• Chinese Techcs.CV

  Spatial density guides faster, safer token pruning in vision-language models

  First-token benchmarks cannot test decode-time eviction, while a shared spatial-density score efficiently prunes visual tokens during both VLM prefill and decode.

  Shanghai Jiao Tong University, Huawei Technologies Ltd · Oct 2026

• Academiccs.LG

  Language models retain near-full accuracy with thirteenfold smaller cache memory

  KV caches compress to one latent per layer block, quantized or evicted, retaining near-full accuracy at 13-14x.

  Rice University · Oct 2026

• Research Labcs.CL

  Precomputing attention indices from earlier layers speeds long-context AI without accuracy loss

  EarlySelectKV precomputes key-value cache selections using earlier hidden states, reducing long-context inference latency without degrading accuracy.

  Institute of Semiconductors, Chinese Academy of Sciences, University of Chinese Academy of Sciences · Nov 2026

• Chinese Techcs.AI

  Language models can compress conversation memory 6.5-fold with little accuracy loss

  Quantizing the full MLA cache to about 2.5 bits per element with role-aware transforms stays nearly lossless on reasoning and coding across four MLA models.

  University of Chinese Academy of Sciences, Institute of Automation, Chinese Academy of Sciences · Oct 2026

• Academiccs.LG

  KV-Surgeon optimizes LLM cache compaction by jointly selecting and compensating for deletions

  Offline KV-cache compaction improves when deletions are scored after surviving entries compensate for them; KV-Surgeon does this and beats prior methods at every tested ratio.

  Zuse Institute Berlin, Technische Universität Berlin · Oct 2026

• Top Universitycs.AIcode

  Recurrent memory signals predict which KV cache entries hybrid LLMs will need

  GateSight uses GDN recurrent memory signals to predict attention KV utility, enabling 90% cache reduction with top accuracy.

  Shanghai Jiao Tong University · Oct 2026

• Big Techcs.LG

  Per-layer KV-cache budgets learned via multi-objective search improve LLM inference efficiency

  MOSAIC automatically learns per-layer KV-cache budgets that outperform uniform allocation and transfer to any target budget, reducing cache by up to 4x.

  Samsung Research, Georgia Institute of Technology · Oct 2026

• AI Startupcs.LG

  Shared KV-caches cut looped-model inference memory without quality loss

  Final-recurrence KV-caches can be reused across recurrences in looped models without quality loss when training induces contractive residual dynamics and corrects first-recurrence errors.

  University of California, San Diego, Together AI · Oct 2026

• Chinese Techcs.CL

  Attention design cuts KV cache and compute while improving LLM quality.

  Virtual Head Attention cuts KV cache, training FLOPs, and improves quality via two linear operations folded into GQA-2.

  Baidu Inc., Peking University · Oct 2026

• Academiccs.AI

  KV-cache eviction method is provably near-optimal, not just heuristic

  DropKV makes KV-cache eviction tractable through independent per-token scoring while preserving near-optimal accuracy and enabling faster, memory-efficient long-context inference.

  University at Albany, SUNY, IBM T. J. Watson Research Center · Oct 2026

• Big Techcs.CL

  Language models can adapt compressed memory without growing their dictionaries

  FAD-KV compresses key-value caches by screening dictionary atoms and replacing inactive ones, keeping memory bounded without compromising quality.

  Rensselaer Polytechnic Institute, IBM Research · Oct 2026

• Chinese Techcs.LG

  Low-rank KV cache compression aligned to attention distributions

  BELT compresses LLM KV caches by up to 70% while improving LongBench accuracy by 23.3 points over the strongest baseline.

  Peking University, JD.com · Oct 2026

• Big Techcs.LG

  Query-adaptive cache reads preserve accuracy while moving fewer bits

  ReadKV uses progressive KV cache codes and query-dependent bit allocation to reduce memory reads while maintaining perplexity within 0.66% of uncompressed.

  Amazon, University of California, Berkeley · Oct 2026

• Academiccs.CL

  Low-precision caches preserve more reasoning history within fixed memory budgets

  Under fixed KV-cache bytes, retaining more low-precision tokens improves long-chain reasoning more than keeping fewer high-precision tokens through eviction alone.

  University of Illinois Urbana-Champaign · Oct 2026

• Chinese Techcs.LG

  Kronecker product factorization compresses key-value caches up to 10x without accuracy loss.

  Kronecker Product Attention compresses KV caches by representing keys and values as Kronecker products of small factors, cutting memory up to 10x while matching or beating prior attention variants.

  Huawei Technologies, University of Waterloo · Oct 2026

• Chinese Techcs.PF

  Local alignment enables predictive coding for error-bounded KV cache compression

  AlignPred compresses persistent KV caches by aligning related blocks and coding residuals under a configurable error budget.

  Hunan University, Tencent · Oct 2026

• Big Techcs.CL

  Refreshing compressed draft caches preserves alignment during long language model outputs

  RIFT compresses draft-model KV caches by mixing prefill and recent tokens, refreshing selection, and training with sparse masks to preserve acceptance length.

  NVIDIA · Oct 2026

• Chinese Techcs.LG

  Joint calibration makes 2-bit latent attention caches more accurate and faster

  RoLaQ jointly calibrates rotation and quantization ranges for 2-bit MLA KV caches, improving accuracy across three MLA models and boosting throughput.

  Harbin Institute of Technology, Tsinghua University · Oct 2026

• Chinese Techcs.CV

  TempoKV compresses video generation KV caches by exploiting key-value predictability asymmetry

  TempoKV predicts keys from history and values from current keys to compress KV cache up to 6.71x for video generation.

  Shanghai Jiao Tong University, South China University of Technology · Oct 2026

• Academiccs.LG

  E8 lattice vector quantization enables 2-bit keys for LLM KV-cache compression

  E8 lattice vector quantization with 2-bit keys and 2-bit values achieves 6x KV-cache compression while preserving near-baseline LLM quality.

  Technical University of Berlin, logsight.ai · Oct 2026

• Big Techcs.AI

  Canonical training improves language models under joint low-bit compression

  CanonQ unifies weight, activation, and KV cache quantization using frozen reference codebooks and joint training.

  Meta AI · Oct 2026

• Independentcs.LG

  Method converts LLMs to efficient attention while handling QKNorm and MoE

  MLAfication converts existing attention architectures to MLA, enabling up to 5.35x throughput and 94.73% KV-cache reduction.

  Oct 2026

• Research Labcs.AR

  Unified retention and compensation for KV cache eviction via coverage calibration

  CORE unifies KV-cache retention and compensation by calibrating one allocation distribution that jointly selects Top-B states and writes latent memory from excluded mass.

  Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences · Oct 2026

• Big Techcs.AI

  Adaptive per-layer choices shrink LLM caches without sacrificing accuracy

  KV-Kaizen learns a per-layer, context-adaptive compression selector along precision, rank, and depth axes, preserving accuracy while reducing cache size by up to 32x.

  Apple · Oct 2026

• Big Techcs.LG

  Adaptive attention pruning compresses language-model caches without sacrificing accuracy

  Universal Attention achieves over 10x KV-cache compression while improving downstream performance.

  IBM Research, SSAIL Lab · Oct 2026

• Big Techcs.LG

  Compressed LLM memories can preserve skills beyond their source documents

  Cartridges++ retains off-context abilities in compressed KV caches with negligible cost.

  Apple · Oct 2026

• Big Techcs.LG

  Prompt-adaptive compression halves transformer KV-cache memory without dropping tokens

  AttSVD compresses the KV cache by storing every token in a per-prompt low-rank subspace, matching dense performance using up to 50% memory.

  Microsoft Applied Sciences Group (ASG) · Oct 2026

• Research Labcs.AI

  Low-rank KV cache compression preserves all tokens during long reasoning

  iS-KV retains all KV-cache positions through synchronized online low-rank compression, preserving reasoning accuracy better than token eviction at matched memory budgets.

  The Hong Kong University of Science and Technology (Guangzhou), Guangdong OPPO Mobile Telecommunications Corp., Ltd. · Oct 2026

• Big Techcs.LG

  Streaming KV caches makes agent context compaction faster to train

  KV-streams streams KV cache across compactions instead of re-prefilling, achieving 2.6x to 5x faster agent training without performance loss.

  Mila, Microsoft · Sept 2026

• Chinese Techcs.LG

  Joint token-feature KV cache compression speeds long-context LLMs

  SlimKV jointly compresses tokens and features in KV cache, achieving high compression with reconstruction-free attention and speedups.

  University of Science and Technology of China, Nanyang Technological University · Oct 2026

• Research Labcs.LGcode

  Training-free inference framework trims redundant computation and memory in looped Transformers

  FlashLoop achieves up to 1.64x speedup and 6x KV-cache reduction in looped transformers without accuracy loss.

  ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems · Oct 2026

• Top Universitycs.LG

  Tailored vector quantization space enables accurate 1-bit KV cache compression

  TaSQ tailors vector quantization for 1-bit KV cache compression, improving accuracy and throughput.

  Seoul National University, Stanford University · Oct 2026

• Academiccs.OS

  Prioritizing action-relevant context cuts LLM agent cache costs

  ActKV retains 98.53% of FullKV accuracy using 25.98% peak KV cache memory, achieving 3.97x token and 3.58x task throughput.

  University of Science and Technology of China, Suzhou Institute for Advanced Research, University of Science and Technology of China · Sept 2026

• Industrycs.LG

  A Vestigial Attention Branch Signals Which Cache Entries to Keep

  On NoPE-MLA models, a 64-dimensional decoupled branch carries a query-independent salience signal that keeps needle retrieval at 1.00 under 8x and 0.96 under 32x compression.

  Yotta Labs · Sept 2026

• Top Universitycs.CV

  Streaming video generators can now output frames at near-watchable speed

  Streaming video generation can emit one finished chunk per concurrent forward round by pipelining denoising stages across model replicas, preserving quality while speeding throughput.

  Shanghai Jiao Tong University, Xi’an Jiaotong University · Nov 2026

• Chinese Techcs.CV

  AI video generators learn which past frames to remember for longer clips

  Long videos generated frame by frame stay coherent when the model learns which past frames to keep in memory instead of following fixed cache rules.

  Tsinghua University, ByteDance Seed · Nov 2026

• Chinese Techcs.RO

  Context scheduling improves long-horizon robot video prediction

  Reorganizing inference context by preserving initial anchor and recent blocks improves long-horizon robot video world model prediction and efficiency.

  The Hong Kong University of Science and Technology (Guangzhou), North China Electric Power University · Oct 2026

• Independentcs.LG

  Temporal indexer prediction speeds up long-context sparse attention by 14.9%

  SPIN predicts which KV cache blocks are important using past scores, reducing indexer overhead by 30-40% and improving throughput by up to 14.9%.

  Oct 2026

• Industrycs.CVcode

  Training-free wrapper accelerates causal video diffusion to 50 FPS without quality loss

  UnStep accelerates causal video diffusion models to 50 FPS on an H100 without quality loss and without retraining.

  Meta Superintelligence Labs · Oct 2026

• Academiccs.LG

  Sparse attention for long-context AI verifies its own quality in one pass

  Coverage-Verified Sparse Attention achieves closed-loop quality control by using the softmax mass fraction to verify and correct sparse attention outputs.

  National Taiwan University · Nov 2026

• Big Techcs.LG

  Million-token prompts served at 2.3 times the throughput without retraining

  Retrofitting pretrained dense-attention large language models with chunk-level sparse attention, without retraining, speeds long-context serving while keeping benchmark scores close to dense attention.

  NVIDIA · Nov 2026

• Industrycs.CL

  Language models can search long contexts without comparing every token

  Zyphra Hierarchical Attention uses a tree of compressed key summaries to achieve subquadratic scaling for attention in large language models.

  Zyphra Research · Nov 2026

• Big Techcs.AI

  Receiver-specific cache sharing cuts multi-agent costs while improving accuracy

  Receiver-conditioned KV-cache filtering cuts multi-agent communication state while improving accuracy and latency over text messaging across several model architectures.

  Columbia University, Amazon Web Services · Sept 2026

• Academiccs.LG

  Language models train faster than Transformers while improving recall

  Muir linear attention uses sensitivity-allocated low-rank gating and matched-filter readout to reduce perplexity and improve long-context recall over Gated DeltaNet-2 and Transformer.

  University of Texas at Austin · Nov 2026

• Big Techcs.CV

  One LiDAR model generates, forecasts and resimulates driving scenes in real time

  A single conditional LiDAR world model unifies generation, forecasting, and resimulation, real-time at 11 FPS on one H100 GPU.

  NVIDIA, University of Illinois, Urbana-Champaign · Oct 2026

• Academiccs.AI

  Bidirectional protocol connects LLM agent workflows with inference engines

  HEAR coordinates agent harness intent with inference-engine state, improving cache scheduling and execution-mode selection while preserving workflow and model semantics.

  Harbin Institute of Technology (Shenzhen) · Oct 2026

• Chinese Techcs.CL

  Coarse-to-fine sparse attention boosts long-context LLM inference speed

  Prism Attention achieves 3.4× decoding speedup over FlashAttention by replacing rigid index reuse with a proposal-refined sparse attention framework.

  Soochow University, Baidu Inc. · Oct 2026

• Independentcs.LG

  Fixed-cost drafter speeds speculative decoding as context lengths grow

  LongSpark decouples drafter cost from prefix length by extracting fixed-size, multiscale views from target verification.

  Oct 2026

• Chinese Techcs.CL

  Two-bit scalar quantization better preserves language model predictions

  Two-bit scalar quantization preserves LLM predictive fidelity through mirrored four-level coding and calibration targeted at teacher-student divergence.

  The Hong Kong Polytechnic University, Huawei Technologies Ltd. · Oct 2026

• Top Universitycs.LG

  Sparse attention mechanism achieves zero-shot extrapolation to longer sequences

  FreeSolo achieves zero-shot length extrapolation with linear-time sparse attention.

  National University of Singapore, University of Toronto · Oct 2026

• Top Universitycs.LG

  End-to-end subspace learning preserves transformers under aggressive low-rank compression

  Learnable Subspace Projections (LSP) learns low-rank compression subspaces jointly for better transformer efficiency.

  Helmholtz Munich, Technical University of Munich · Oct 2026

• Chinese Techcs.AI

  Search framework finds better layer-wise settings for efficient LLM inference

  InferOpt replaces mechanism-specific heuristics with a unified search framework for LLM inference configurations.

  Huawei Technologies Co., Ltd. · Oct 2026

• Independentcs.AI

  Language models can gain accuracy by adjusting precision during generation

  AutoEMatic dynamically adjusts LLM inference precision at each step, improving accuracy by up to 21.7% over static NVFP4 with minimal throughput loss.

  Nov 2026

• Independentcs.CL

  Sparse attention trained on 32K tokens works at million-token contexts

  RecSA achieves million-token length extrapolation with logarithmic compute cost using a tree-structured KV cache.

  Nov 2026

• Research Labcs.CV

  Video AI models generate clips with 99% fewer attention computations

  ESTRA accelerates video diffusion models up to 5.39× by computing only 1% of attention exactly.

  The University of Sydney, CSIRO · Nov 2026

• Independentcs.AI

  Cached reasoning traces match frontier LLMs at a fraction of the cost

  Agentic reasoning caching converts LLM reasoning traces into CPU-executable caches that perform on par with frontier LLMs at drastically lower cost.

  Nov 2026

• Independentcs.CL

  Safety checks inside a model can flag harmful output as it streams

  A guardrail built into a large language model's own internal signals detects unsafe text as it is generated, with little added serving cost.

  Nov 2026

• Independentcs.AI

  Clustering keys makes 10-million-token attention 22x faster with similar accuracy

  Nov 2026

• Independentcs.AI

  Language models process million-token prompts up to 1.8 times faster

  Routing each query through a causal inverted-file index over indexer keys recovers more of the exhaustive top-k than block-based selection at higher prefill throughput.

  Nov 2026

• Top Universitycs.LG

  Language models can generate text 2.52 times faster using compression

  MINE uses lossless entropy coding to accelerate single-stream LLM inference by 2.52× on an RTX 4060 Ti.

  ETH Zurich · Nov 2026

• Top Universitycs.CV

  Video generators can run twice as fast without retraining

  Accurate Block Attention (AB-Attention) skips 89-92% of attention in video diffusion transformers while improving PSNR by up to 4.4 dB.

  InfRec, The Chinese University of Hong Kong, Shenzhen · Oct 2026

• Big Techcs.CL

  On-device AI drafters can accept more tokens with less memory by separating context from local attention

  HyFlash achieves higher acceptance than full-context attention with less KV storage, especially for shallow drafters and long contexts.

  Microsoft Applied Sciences Group · Nov 2026

• Academiccs.NI

  Avoiding routing collisions cuts cache-transfer delays in AI inference clusters

  ARK avoids routing collisions for KV cache transfers in disaggregated LLM inference by coordinating source port selection to spread flows across distinct spines.

  Georgia Institute of Technology · Nov 2026

• Chinese Techcs.LG

  Aligning FP4 training and rollout stabilizes reinforcement learning

  TRACE aligns FP4 training and rollout quantization for MoE RL, matching BF16 rollout performance while accelerating rollout by up to 5.4x.

  Alibaba Token Hub, Alibaba Group, Ohio State University · Oct 2026

• Chinese Techcs.CL

  Query-side adapters recover full-context quality in KV cache reuse

  Query-side adaptation of attention projections enables KV cache reuse that matches full prefill quality without recomputation, achieving up to 3.73x speedup.

  Eastern Institute of Technology, Ningbo, The Hong Kong Polytechnic University · Sept 2026

• Big Techcs.LG

  Auctions can prioritize LLM requests while preserving cache efficiency

  An inference auction where users bid for priority increases welfare without harming latency or cache efficiency.

  University of California, Berkeley, Toyota Technological Institute at Chicago · Oct 2026