The authors compared scalar and vector post-training quantization, which compresses an already trained model without full retraining.
Vector codebooks preserve useful language models at roughly two bits
Across six models, vector quantization retained most question-answering accuracy while scalar two-bit methods collapsed to chance.
Big Tech
Rhea Sanjay Sukthanker · Riccardo Grazzi · James Hensman · Pashmina Cameron
Applied Sciences Group (ASG), Microsoft · Microsoft Research
Research Digest··3 min read
8 billion to 32 billion parameters.
§