← All concepts

vector quantization

Vector quantization means replacing a block of many numbers with a single code from a small, pre-made lookup table, instead of rounding each number individually. The table holds typical patterns, so one short code can capture a lot of information. This shrinks memory use, which matters for large language models that store lots of intermediate values while generating text.

The papers show that vector quantization consistently beats simple per-number rounding at very low bit rates. It works for compressing model weights, the key-value cache used during generation, and even speech embeddings, often matching full-precision quality while using far less memory and allowing larger batch sizes. A recurring theme is that careful design choices, like how codes are grouped or trained, matter more at 2 bits than at 4 bits, and that proper joint training can make these extreme compressions nearly lossless. It can also change model behavior in unexpected ways, such as reducing hallucination in vision-language models.

Papers
11
Released code
0
First seen
Oct 2026
Latest
Nov 2026

11 papers in the last two months, against 0 in the two before.

Start here

Three papers to read first, in the order they came out
  1. 1Vector codebooks preserve useful language models at roughly two bits

    At 2-bit weights, vector codebooks with jointly chosen calibration and recovery data keep 89-97% of full-precision QA accuracy where scalar PTQ fails.

    Oct 2026
  2. 2Quantization error doesn't determine semantic fidelity of discrete visual tokens; bit budget does

    Discrete visual tokens with large enough bit budgets can match continuous representations in semantic fidelity.

    Oct 2026
  3. 3AI search averages 1.8 times the query throughput of rivals

    Ginger is a C++ library that speeds up approximate nearest neighbor search by 1.8x on average while using less memory.

    Nov 2026

Who is working on it

Microsoft Research 2

The papers

Most central to this idea first, not most recent.

Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.

vector quantization · Research concepts | Zotpaper