Running a large language model on a phone or drone is hard because the model's weights, the numbers that store its learned knowledge, take up too much memory and take too long to move from memory to the processor. Each time the model generates a word, it must read all those numbers from memory, and moving them is slow and uses energy. Traditional methods store each weight with the same number of bits, even though some numbers appear much more often than others, wasting space.
LAZYWEIGHTS solves this by first shaping the weights during quantization, the process of squeezing them into fewer bits, to make common numbers even more common. Then it uses Huffman coding, a technique that assigns short codes to frequent numbers and longer codes to rare ones, just like Morse code gives short signals to common letters. A tiny decoder chip, smaller than a grain of rice, decompresses the codes on the fly as the model reads them from memory.
On a Jetson Orin Nano Super, this approach cuts energy per generated word by up to 38% and speeds it up by up to 36%. On a Jetson Nano, a 7-billion-parameter model that previously couldn't fit now runs 6.6 times faster than before. Models in common formats become 5.7 to 31.4 percent smaller without any loss in quality.