vector quantization
Vector quantization means replacing a block of many numbers with a single code from a small, pre-made lookup table, instead of rounding each number individually. The table holds typical patterns, so one short code can capture a lot of information. This shrinks memory use, which matters for large language models that store lots of intermediate values while generating text.
The papers show that vector quantization consistently beats simple per-number rounding at very low bit rates. It works for compressing model weights, the key-value cache used during generation, and even speech embeddings, often matching full-precision quality while using far less memory and allowing larger batch sizes. A recurring theme is that careful design choices, like how codes are grouped or trained, matter more at 2 bits than at 4 bits, and that proper joint training can make these extreme compressions nearly lossless. It can also change model behavior in unexpected ways, such as reducing hallucination in vision-language models.
- Papers
- 11
- Released code
- 0
- First seen
- Oct 2026
- Latest
- Nov 2026
11 papers in the last two months, against 0 in the two before.
Start here
Three papers to read first, in the order they came out- 1Vector codebooks preserve useful language models at roughly two bits
At 2-bit weights, vector codebooks with jointly chosen calibration and recovery data keep 89-97% of full-precision QA accuracy where scalar PTQ fails.
Oct 2026 - 2Quantization error doesn't determine semantic fidelity of discrete visual tokens; bit budget does
Discrete visual tokens with large enough bit budgets can match continuous representations in semantic fidelity.
Oct 2026 - 3AI search averages 1.8 times the query throughput of rivals
Ginger is a C++ library that speeds up approximate nearest neighbor search by 1.8x on average while using less memory.
Nov 2026
How fast it is moving
Often studied with
quantization-aware trainingTraining a neural network with simulated quantization in the forward pass so its weights and activations adapt to low-precision formats, improving deployment accuracy.4 togetherkv cache compressionReducing the size of the key-value cache in transformers via selective retention, low-rank updates, or adaptive compression.3 togetherlattice quantizationQuantization that maps groups of weights or values to the nearest point in a regular lattice, like E8, enabling compact codes without a codebook.3 togetherWho is working on it
The papers
Most central to this idea first, not most recent.
- Big Techcs.IR
AI search averages 1.8 times the query throughput of rivals
Ginger is a C++ library that speeds up approximate nearest neighbor search by 1.8x on average while using less memory.
Google Research, KIT · Nov 2026
- Top Universitycs.CV
Quantization error doesn't determine semantic fidelity of discrete visual tokens; bit budget does
Discrete visual tokens with large enough bit budgets can match continuous representations in semantic fidelity.
Huazhong University of Science and Technology, Meituan LongCat · Oct 2026
- Big Techcs.AI
Vector codebooks preserve useful language models at roughly two bits
At 2-bit weights, vector codebooks with jointly chosen calibration and recovery data keep 89-97% of full-precision QA accuracy where scalar PTQ fails.
Applied Sciences Group (ASG), Microsoft, Microsoft Research · Oct 2026
- Chinese Techcs.LG
New method compresses large language models to 2 bits with minimal accuracy loss
TVQ achieves near-lossless 2-bit quantization of LLMs like LLaMA2-70B with only 1% accuracy drop.
Huawei Technologies Co. Ltd. · Nov 2026
- Academiccs.LG
Language models can be trained for compression nearly 13 times faster
SQuAT improves low-bit LLM compression by applying separate straight-through estimators for shape and gain in vector quantization.
University of Copenhagen, Qualcomm AI Research · Oct 2026
- Academiccs.LG
E8 lattice vector quantization enables 2-bit keys for LLM KV-cache compression
E8 lattice vector quantization with 2-bit keys and 2-bit values achieves 6x KV-cache compression while preserving near-baseline LLM quality.
Technical University of Berlin, logsight.ai · Oct 2026
- Chinese Techcs.AI
E8 lattice quantization preserves model accuracy at two-bit precision
E8-QAT delivers accurate 2-bit LLM weights by projecting eight-weight groups onto an implicit E8 lattice without explicit codebook storage or search.
Peking University, City University of Hong Kong · Oct 2026
- Academic
VQ-tokenized VLMs share a circuit that causes object hallucinations
Object hallucination in VQ-tokenized vision-language models is caused by an early-layer attention routing circuit installed by vector quantization, not by generic miscalibration.
Adobe, Arizona State University · Oct 2026
- Big Techcs.AI
Canonical training improves language models under joint low-bit compression
CanonQ unifies weight, activation, and KV cache quantization using frozen reference codebooks and joint training.
Meta AI · Oct 2026
- Top Universitycs.LG
Tailored vector quantization space enables accurate 1-bit KV cache compression
TaSQ tailors vector quantization for 1-bit KV cache compression, improving accuracy and throughput.
Seoul National University, Stanford University · Oct 2026
- Big Techcs.CL
Simple training method stops speech-LLMs from ignoring new instructions
DirectSpeech2LLM preserves an LLM's instruction-following ability on unseen speech tasks without task-specific training.
IIIT Delhi, Microsoft Research · Oct 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.