Agentic LLM inference accumulates long KV caches across iterative observation-reasoning-action loops, causing memory overhead and limiting throughput. Existing compression methods treat all tokens equally, but actions are more important for task progress. ActKV is the first framework tailored for agentic LLM inference, using three key ideas: action-oriented eviction that exploits stable action access patterns to retain future-critical entries, confidence-driven adaptive budget allocation that adjusts memory based on the LLM's own confidence, and page-aware compression management that standardizes compression into efficient kernels. On long-trace tasks, ActKV retains an average of 98.53% of FullKV's accuracy with only 25.98% of its peak KV cache memory, while achieving 3.97 times token throughput and 3.58 times task throughput, demonstrating state-of-the-art performance.
Prioritizing action-relevant context cuts LLM agent cache costs
ActKV preserved 98.53% of full-cache accuracy while using 25.98% of peak cache memory on long-trace agent tasks.
Academic
Zihan Wang · Cheng Tang · Lei Gong · Chao Wang · Wenqi Lou · Teng Wang · +1 more
University of Science and Technology of China · Suzhou Institute for Advanced Research, University of Science and Technology of China
Research Digest··2 min read
In one line
ActKV retains 98.53% of FullKV accuracy using 25.98% peak KV cache memory, achieving 3.97x token and 3.58x task throughput.
Before you read this
Three ideas this paper builds onkv cache compressionReducing the size of the key-value cache in transformers via selective retention, low-rank updates, or adaptive compression.79 papers →agent memoryA repository where an agent stores past interactions and outcomes to inform future decisions.232 papers →confidence calibrationConfidence calibration aligns a model's predicted confidence with its actual accuracy across predictions.86 papers →
Read it as
Wang and colleagues introduce ActKV, a cache-compression system designed around the actions generated by LLM agents rather than the quality of every generated token.
The authors studied key-value (KV) cache growth during iterative observation, reasoning, and action loops.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.