post-training quantization
Post-training quantization shrinks a trained neural network by storing its weights (and sometimes activations) in fewer bits, instead of the usual 32- or 16-bit numbers. This cuts memory use and speeds up inference, without retraining the model. The catch is that aggressive shrinking, especially to 2 or 3 bits per weight, can destroy accuracy. Recent work therefore treats quantization as a delicate engineering problem: how to round numbers, which calibration data to use, and how to keep errors from compounding across layers.
Across these papers, the common thread is that accuracy at very low bit-widths depends less on raw compression and more on how errors are managed. Methods simulate noise from upstream layers, choose bit-widths per tensor based on sensitivity, refine quantized weights against worst-case shifts, or co-design the quantizer with the hardware. Several show that 2-bit quantization can become practical when calibration data, recovery data, and representations are chosen jointly, cutting memory and compute while preserving most of the original model's quality.
- Papers
- 54
- Released code
- 5
- First seen
- Oct 2026
- Latest
- Nov 2026
54 papers in the last two months, against 0 in the two before.
Start here
Three papers to read first, in the order they came out- 14-bit language models run as integer math on Apple's M5 chips without extra accuracy loss
Standard MXFP4 weight rows fold into one exact INT8 vector times a power-of-two scale, a condition real weights already meet, so 4-bit models run on INT8 matrix units.
Nov 2026 - 2Combining model compression techniques can collapse small language models
INT8 quantization is lossless; NF4 with pruning collapses models; LoRA recovery works without a teacher.
Nov 2026 - 3Language models can gain accuracy by adjusting precision during generation
AutoEMatic dynamically adjusts LLM inference precision at each step, improving accuracy by up to 21.7% over static NVFP4 with minimal throughput loss.
Nov 2026
How fast it is moving
Often studied with
large language modelA neural network trained on massive text corpora to generate coherent text and perform diverse language tasks.10 togethermixture of expertsA neural architecture with many specialized sub-networks, only a few activated per input via a router, balancing capacity and efficiency.10 togetherquantization-aware trainingTraining a neural network with simulated quantization in the forward pass so its weights and activations adapt to low-precision formats, improving deployment accuracy.10 togetherWho is working on it
The papers
Most central to this idea first, not most recent.
- Independentcs.AI
Language models can gain accuracy by adjusting precision during generation
AutoEMatic dynamically adjusts LLM inference precision at each step, improving accuracy by up to 21.7% over static NVFP4 with minimal throughput loss.
Nov 2026
- Big Techcs.AI
Combining model compression techniques can collapse small language models
INT8 quantization is lossless; NF4 with pruning collapses models; LoRA recovery works without a teacher.
Amazon Web Services · Nov 2026
- Top Universitycs.LGcode
4-bit language models run as integer math on Apple's M5 chips without extra accuracy loss
Standard MXFP4 weight rows fold into one exact INT8 vector times a power-of-two scale, a condition real weights already meet, so 4-bit models run on INT8 matrix units.
UT Southwestern Medical Center, Emory University · Nov 2026
- Top Universitycs.LG
Language models can lag 50 to 150 updates behind training
Standard bf16 rounding silently discards most RL updates, creating a hidden lag that unbiased rounding eliminates.
Stanford University · Nov 2026
- Independentcs.LGcode
Lower precision beats structured pruning until GPU quantization limits are reached
Adding 2:4 sparsity is worse than dropping to lower precision when available, but at the hardware floor it provides speedup with retraining.
Oct 2026
- Academiccs.CR
Signed LLM weights do not certify the quantized files users actually run
Signing full-precision LLM weights does not verify deployed quantized models unless the transformation inputs and build environment support bit-exact reproduction.
National Yang Ming Chiao Tung University, National Taiwan University of Science and Technology · Oct 2026
- Big Techcs.AI
Vector codebooks preserve useful language models at roughly two bits
At 2-bit weights, vector codebooks with jointly chosen calibration and recovery data keep 89-97% of full-precision QA accuracy where scalar PTQ fails.
Applied Sciences Group (ASG), Microsoft, Microsoft Research · Oct 2026
- Chinese Techcs.LG
Low-precision training of AI expert models needs matching fixes on both sides
Symmetric quantization recovery closes the accuracy gap in FP4 reinforcement learning for mixture-of-experts LLMs.
Huawei · Nov 2026
- Big Techcs.AI
Scalar two-bit weight quantization can preserve language model reasoning accuracy
Strictly scalar two-bit quantization with sensitivity-based bit allocation, codeword search, scale recalibration, and reasoning-based selection preserves LLM reasoning.
Microsoft Applied Sciences Group · Oct 2026
- Chinese Techcs.AI
Language models can compress conversation memory 6.5-fold with little accuracy loss
Quantizing the full MLA cache to about 2.5 bits per element with role-aware transforms stays nearly lossless on reasoning and coding across four MLA models.
University of Chinese Academy of Sciences, Institute of Automation, Chinese Academy of Sciences · Oct 2026
- Research Labcs.AI
4-bit compression speeds up LLM training by 56 percent with minimal accuracy loss
FP4Quant compresses model-parallel LLM communication to 4-bit floats, cutting quantization overhead and improving end-to-end training throughput by up to 1.56x.
University of Chinese Academy of Sciences, Institute of Computing Technology, Chinese Academy of Sciences · Nov 2026
- Academiccs.AR
Mixed-precision LLM accelerator turns lower bit weights into proportionally faster compute
FIGFLEX's radix-4 FP-INT datapath makes LLM GEMM compute scale with weight bit-width, and its mixed-precision quantizer beats existing low-bit methods.
University of California, Irvine · Oct 2026
- Big Techcs.PL
One program generates fast neural network kernels for most quantized formats
Separating value representation from algorithm and schedule generates tuned kernels for dozens of quantized format pairs from a single routine.
Adobe Research · Nov 2026
- Industrycs.SD
Open-vocabulary speech recognition runs on a $5 microcontroller without a neural accelerator
Oído brings open-vocabulary English speech recognition to a $5 microcontroller without a neural accelerator.
Lokutor · Oct 2026
- Industrycs.CL
One random probe can guide data-free mixed-precision model quantization
Round-to-nearest quantization error is spectrally flat, letting one Gaussian probe estimate tensor sensitivity and allocate mixed precision without calibration data.
Blacksheep ai. · Oct 2026
- Chinese Techcs.LG
Two-rate precision allocation cuts MoE memory footprint and decode traffic
DuoRate stores MoE weights once and allocates per-expert read precision per workload, matching 4-bit formats at 2.9 bpw on Qwen3-30B-A3B.
Huawei Technologies Ltd. · Oct 2026
- Top Universitycs.AI
Lower reconstruction loss can degrade quantized LLM performance; robust refinement fixes it
Distributionally robust quantization (DRQ) minimizes worst-case reconstruction loss under constrained activation shifts, avoiding the calibration-loss trap.
University of Science and Technology of China, University College London · Oct 2026
- Independentcs.LGcode
EntroPack compresses neural weights at finely adjustable storage rates
EntroPack entropy-codes row-normalized E8 lattice quantized weights to hit arbitrary bitrates with lower error than fixed-width formats and fast GPU reconstruction.
Oct 2026
- Academiccs.LGcode
Neural network precision floor follows a depth law predicted by amplification
The precision floor of neural networks follows a power law in depth, predicted by predictive amplification with a constant collapse constant.
Mutah University · Oct 2026
- Top Universitycs.CV
Centering activations enables calibration-free 4-bit diffusion transformer quantization
Exact per-token mean centering enables calibration-free 4-bit DiT quantization to match calibrated SVDQuant across three model families.
Tenstorrent, EPFL · Oct 2026
- Big Techcs.CV
Recalibrating activations helps quantized vision transformers withstand distribution shifts
QuAR recalibrates activations at frozen quantizer inputs to correct distribution shift, achieving top accuracy among backprop-free TTA methods across bit widths.
KAIST, Samsung AI Center-Cambridge · Oct 2026
- Chinese Techcs.CL
Two-bit scalar quantization better preserves language model predictions
Two-bit scalar quantization preserves LLM predictive fidelity through mirrored four-level coding and calibration targeted at teacher-student divergence.
The Hong Kong Polytechnic University, Huawei Technologies Ltd. · Oct 2026
- Chinese Techcs.LG
H-SVDQuant improves 4-bit quantization of diffusion transformers with low-rank branches
H-SVDQuant achieves rank-4 W4A4 quantization that surpasses rank-32 SVDQuant on diffusion transformers and LLMs.
The University of Hong Kong, Huawei Technologies Co., Ltd. · Oct 2026
- Big Techcs.LG
Noise-robust layers narrow the accuracy gap in parallel LLM quantization
Parallel LLM quantization with activation denoising recovers much of sequential quantization's accuracy benefit without serial overhead.
Munich Data Science Institute, Technical University of Munich · Oct 2026
- Industrycs.CRcode
Language model proofs take 41 to 50 times serving time
Sable proves, on a single GPU, that a committed integer-quantized language model produced a specified token sequence exactly.
MoonMath.ai · Nov 2026
- Big Techcs.AI
Larger language models can lose more retrieval ability after compression
Larger language models are more vulnerable to compression because their internal compensation for damaged circuits weakens as scale increases.
Indraprastha Institute of Information Technology, Delhi, TurboML · Nov 2026
- Big Techcs.AI
Affine folding cuts KV cache decoding latency by up to 3.4x on Hopper and Blackwell GPUs
FOSCAR speeds attention over compact caches by moving shared quantization corrections out of per-value reconstruction while preserving the exact result.
FutureMLS Lab, Microsoft Research · Nov 2026
- Chinese Techcs.LG
Language models can shrink fact-storage memory nearly eightfold, preserving most facts
For quantized conditional memory, per-lookup gate values, not access frequency, should decide which rows get more bits; gate-based selection beats frequency and random.
Institute of Automation, Chinese Academy of Sciences, Tencent · Oct 2026
- Chinese Techcs.LG
New method compresses large language models to 2 bits with minimal accuracy loss
TVQ achieves near-lossless 2-bit quantization of LLMs like LLaMA2-70B with only 1% accuracy drop.
Huawei Technologies Co. Ltd. · Nov 2026
- Big Techcs.CV
Video generators can nearly halve history storage while improving reconstruction quality
A compression scheme for cached attention state in autoregressive video models preserves generation quality better by tuning Keys to match downstream outputs rather than to reconstruct the raw cache.
Zhejiang University, NVIDIA · Nov 2026
- Big Techcs.CL
AI coding model beats predecessor using roughly one-fifth the active parameters
Nemotron 3.5 Super trains a 12B-active multimodal model that exceeds a 4.6x larger model on SWE-bench Pro.
NVIDIA · Oct 2026
- Top Universitycs.AI
Compression preserves broad LLM skills while weakening context-sensitive abilities
Compression preserves coarse abilities while degrading fine-grained, context-dependent ones, and quantization retains abilities better than pruning.
City University of Hong Kong, Tongji University · Oct 2026
- Top Universitycs.LG
Compressed looped models settle, not drift, so precise final loops recover them
Quantization errors in settling looped models shift their fixed point rather than accumulate, allowing collapse prediction and recovery with a few higher-precision loops.
Carnegie Mellon University, ML Collective · Oct 2026
- Academiccs.CL
Compression Widens Demographic Fairness Gaps in Whisper Speech Recognition Models
Post-training pruning and quantization can sharply amplify demographic error burdens in Whisper, while distillation usually narrows them.
The Ohio State University, Air Force Research Laboratory · Oct 2026
- Chinese Techcs.LG
Joint calibration makes 2-bit latent attention caches more accurate and faster
RoLaQ jointly calibrates rotation and quantization ranges for 2-bit MLA KV caches, improving accuracy across three MLA models and boosting throughput.
Harbin Institute of Technology, Tsinghua University · Oct 2026
- Independentcs.LG
Tetra serves 2.7-bit language models without expanding lattice codes
Tetra serves Leech-lattice-quantized Qwen3 models at about 2.7 bits per parameter by decoding compact codes inside matrix-vector multiplication.
Oct 2026
- Independentcs.DC
Fusing adapters into low-precision GEMMs cuts latency for AI serving and training
EpiLoRA fuses LoRA adapters into low-precision GEMM kernels, cutting serving latency and enabling RL training on FP8 weights.
Nov 2026
- Top Universitycs.LG
A single attack triggers malicious behavior in LLMs across multiple deployment-time transformations
An attack creates LLMs that appear benign but become malicious when quantization, pruning, or merging is applied.
ETH Zurich · Oct 2026
- Big Techcs.AI
Language models can improve by learning from their weakened copies
Contrasting a model with its weakened copy improves reasoning accuracy without privileged information.
Massachusetts Institute of Technology, NVIDIA · Oct 2026
- Chinese Techcs.LG
Multi-precision LLM weight format keeps a single prefix-readable stream
StagQ stores LLM weights in a multi-precision format where any prefix is a valid lower-precision code, improving accuracy at low bits.
Huawei Technologies Ltd. · Oct 2026
- Academiccs.LG
New method makes quantized AI models up to 31% smaller with no quality loss
LAZYWEIGHTS tunes GPTQ quantized weights for Huffman compression, reducing edge LLM memory footprint by 5.7–31.4% at equal perplexity.
University of Texas at Austin, Case Western Reserve University · Nov 2026
- Chinese Techcs.CV
Video assistants need to track event progress to follow conditional instructions
Streaming VLMs fail compositional instructions like 'notify when pot boils after stove on and person leaves'; TEMPI agent achieves 44.4% success.
Hong Kong University of Science and Technology (Guangzhou), Tsinghua University · Nov 2026
- Big Techcs.CL
Small language model matches larger ones on conversational tasks with half the memory
Nemotron 3.5 Bolt E4B 4GB achieves the highest average accuracy and fewest output tokens for fast conversational agents.
NVIDIA · Nov 2026
- Top Universitycs.CV
Small vision-language AI models can be compressed fourfold with only 1% accuracy loss
ProQuant makes low-bit W4A8 quantization of small multimodal LLMs nearly lossless by progressively restoring multimodal understanding before language modeling.
The Hong Kong University of Science and Technology, Northwestern Polytechnical University · Nov 2026
- Top Universitycs.RO
Robot AI speed depends more on action generation than model size
Action-generation strategy determines VLA inference latency on edge hardware more than model size.
National University of Singapore, 0G Labs · Oct 2026
- Independentcs.LG
Recurrent-state quantization fails at long decode horizons due to scale-state feedback
Per-token dynamic scaling of recurrent state quantization fails over long decode due to scale-state feedback; persistent scaling outperforms.
Independent Researcher · Oct 2026
- Big Techcs.AI
Verified retries let smaller code models recover lost capability
Verified retries from smaller models can match or beat much larger models on code benchmarks while using less memory, time, and energy.
Texas A&M, Apple · Oct 2026
- AI Startupcs.LG
Training models for deployment preserves quality under four-bit quantization
QATFactory enables quantization-aware training that improves deployed LLM quality over post-training quantization for NVFP4, MXFP4, and Q4_K formats.
Together AI, University of Illinois Urbana-Champaign · Oct 2026
- Big Techcs.LG
Training-free quantization halves memory traffic for linear attention states
LeapQuant quantizes recurrent states to 8-bit with near-lossless accuracy, reducing memory and speeding inference in linear attention models.
UC Berkeley, University of Washington · Oct 2026
- Big Techcs.CL
Training on self-generated answers improves reasoning in low-bit language models
OnQ recovers reasoning in low-bit LLMs by training quantized students on their own generated responses with candidate-wise Dynamic KL, outperforming fixed-response distillation.
University of Michigan, Ann Arbor, University of California, Santa Barbara · Oct 2026
- Chinese Techcs.AI
Open-MMUnlearning framework unifies MLLM unlearning evaluation across five benchmarks and eight models.
Open-MMUnlearning unifies benchmarks, models, and methods for standardized MLLM unlearning evaluation.
Institute of Automation, Chinese Academy of Sciences, University of the Chinese Academy of Sciences · Oct 2026
- Big Techcs.AR
Generating MoE expert weights on chip reduces inference memory pressure
FORGE regenerates MoE expert weights on-chip from compact rotor parameters, replacing HBM weight loads and improving throughput 1.3-2.8x with some accuracy loss.
University of Wisconsin-Madison, Korea University · Oct 2026
- AI Startupcs.LG
On-policy residual training helps diffusion language models decode faster
Residual context for diffusion language models must be learned on-policy with an entropy-contraction objective to improve speed and quality.
Together AI, University of Illinois Urbana-Champaign · Oct 2026
- Independentcs.LG
Method converts LLMs to efficient attention while handling QKNorm and MoE
MLAfication converts existing attention architectures to MLA, enabling up to 5.35x throughput and 94.73% KV-cache reduction.
Oct 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.