Running a large language model on your own computer is slow because every weight must be fetched from memory for each word generated, and you can't combine multiple requests. This bottleneck limits local and low-latency use.
The authors present MINE, a method that compresses model weights using a kind of lossless data compression adapted for GPUs. It exploits the fact that weights are not all equally likely, so they can be stored with fewer bits on average. The system auto-tunes compression for each model layer and GPU model to find the best speed for a given memory budget.
On an RTX 4060 Ti, MINE made the model run 2.52 times faster than existing methods while keeping the same accuracy. This means bigger models can fit in smaller GPUs, or responses can be faster on local hardware.