Affine folding cuts KV cache decoding latency by up to 3.4x on Hopper and Blackwell GPUs

FOSCAR replaces per-element dequantization with group-level affine operations, doubling throughput in long-context serving.

Big Tech

Donglin Zhuang · Pragaash Ponnusamy · Xiaoxia Wu · Shuaiwen Leon Song · Zhongzhu Zhou

FutureMLS Lab · Microsoft Research

Research Digest·9 November 2026·3 min read

Illustration: Affine folding cuts KV cache decoding latency by up to 3.4x on Hopper and Blackwell GPUs

In one line

FOSCAR speeds attention over compact caches by moving shared quantization corrections out of per-value reconstruction while preserving the exact result.

Before you read this

Three ideas this paper builds on

kv cache compressionReducing the size of the key-value cache in transformers via selective retention, low-rank updates, or adaptive compression.73 papers →large language modelA neural network trained on massive text corpora to generate coherent text and perform diverse language tasks.290 papers →post-training quantizationQuantizing pre-trained neural network weights and activations to48 papers →

Read it as

When a language model generates text one token at a time, it repeatedly rereads stored information about earlier tokens. Compressing this cache saves memory traffic, but unpacking and correcting its values can consume enough processor work to erase the gain.

FOSCAR uses affine folding: each group’s shared scale, a multiplier, and offset, a value added, are applied to other parts of the calculation rather than to every cached value. It also prepares the packed values for specialized graphics-processor matrix hardware.

The method preserves the exact attention result in exact arithmetic. In long-context serving tests, peak throughput rose by up to 170% and 341% against the two reported cache baselines. That means more requests can be served without the cache preparation becoming the bottleneck.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

kv cache compression large language model post-training quantization long-context processing attention mechanism