Native FP4 KV cache keeps long-context LLM accuracy while doubling throughput

BitDecoding-FP4 reduces query logit error from 60% to 1% and prevents probability underflow, enabling B300 serving at 2.4–3.0× the peak throughput of BF16 KV.

Independent

Dayou Du · Woon Yee Ng · Guangming Sheng · Nick Lau · Jianyi Cheng · Luo Mai

Research Digest·9 November 2026·3 min read

Thread:Low-Bit Quantisation

Illustration: Native FP4 KV cache keeps long-context LLM accuracy while doubling throughput

In one line

BitDecoding-FP4 enables accurate native FP4 KV cache decoding, achieving up to 3.0x throughput over BF16 KV.

Before you read this

Two ideas this paper builds on

kv cache compressionReducing the size of the key-value cache in transformers via selective retention, low-rank updates, or adaptive compression.73 papers →quantization-aware trainingTraining a neural network with simulated quantization in the forward pass so its weights and activations adapt to low-precision formats, improving deployment accuracy.23 papers →

Read it as

Large language models (LLMs) generate text one token at a time, and each step must read a growing memory cache of previous tokens. This cache becomes a bottleneck for speed and memory, especially with long contexts like entire books.

BitDecoding-FP4 shrinks this cache by storing values in a very compact 4-bit format and uses a co-design of new algorithms and hardware-aware kernels to keep accuracy high. It reduces three main sources of error: quantizing queries, avoiding underflow in attention probabilities, and handling outliers in keys.

The system stays within 0.5 percent of the original accuracy while delivering 2.4 to 3.0 times higher peak serving throughput than standard 16-bit cache, enabling faster and more memory-efficient deployment of large models with long contexts.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

kv cache compression quantization-aware training