← All concepts

post-training quantization

Post-training quantization shrinks a trained neural network by storing its weights (and sometimes activations) in fewer bits, instead of the usual 32- or 16-bit numbers. This cuts memory use and speeds up inference, without retraining the model. The catch is that aggressive shrinking, especially to 2 or 3 bits per weight, can destroy accuracy. Recent work therefore treats quantization as a delicate engineering problem: how to round numbers, which calibration data to use, and how to keep errors from compounding across layers.

Across these papers, the common thread is that accuracy at very low bit-widths depends less on raw compression and more on how errors are managed. Methods simulate noise from upstream layers, choose bit-widths per tensor based on sensitivity, refine quantized weights against worst-case shifts, or co-design the quantizer with the hardware. Several show that 2-bit quantization can become practical when calibration data, recovery data, and representations are chosen jointly, cutting memory and compute while preserving most of the original model's quality.

Papers
54
Released code
5
First seen
Oct 2026
Latest
Nov 2026

54 papers in the last two months, against 0 in the two before.

Start here

Three papers to read first, in the order they came out
  1. 14-bit language models run as integer math on Apple's M5 chips without extra accuracy loss

    Standard MXFP4 weight rows fold into one exact INT8 vector times a power-of-two scale, a condition real weights already meet, so 4-bit models run on INT8 matrix units.

    Nov 2026
  2. 2Combining model compression techniques can collapse small language models

    INT8 quantization is lossless; NF4 with pruning collapses models; LoRA recovery works without a teacher.

    Nov 2026
  3. 3Language models can gain accuracy by adjusting precision during generation

    AutoEMatic dynamically adjusts LLM inference precision at each step, improving accuracy by up to 21.7% over static NVFP4 with minimal throughput loss.

    Nov 2026

Who is working on it

NVIDIA 4Institute of Automation, Chinese Academy of Sciences 3Huawei Technologies Ltd. 3Microsoft Research 2University of Chinese Academy of Sciences 2Tsinghua University 2

The papers

Most central to this idea first, not most recent.

Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.