Large language models, like the one powering ChatGPT, are so big they require huge amounts of memory, making them expensive to run. A compression method called shrinks the model by replacing groups of numbers with representative vectors, but extreme compression usually hurts accuracy, and past methods couldn't fix that effectively.
The authors developed TVQ, which makes the compression process trainable using straight-through estimation, a trick that allows gradient descent, a standard machine learning optimization, to improve the compressed model block by block. This approach works with existing vector quantization schemes and is easy to implement in tools like PyTorch.
TVQ compressed a 70-billion-parameter model to just 2 bits per weight with only a 1% drop in accuracy on question-answering benchmarks. This shows large models can be made much smaller and faster to run without significant loss of quality, which is crucial for deploying them on limited hardware.