Deploying small language models on limited hardware requires shrinking techniques like quantization (making numbers smaller) and pruning (removing weak connections). But engineers lack clear guidance on which combinations or order to use, often relying on trial and error.
This study systematically tested 72 combinations of quantization, pruning, and recovery fine-tuning on four model sizes. They also tested whether the order of applying these techniques matters.
The key results: using 8-bit quantization alone cuts memory by 40-45% with no accuracy loss. But combining a more aggressive 4-bit type (NF4) with pruning completely broke every model tried. The good news: damaged models can recover 97% of lost quality using LoRA fine-tuning, which does not need a separate teacher model. This gives engineers a clear, simple strategy: use 8-bit quantization for safe memory savings, and if pruning is needed, fix it with LoRA, avoiding the broken combination.