Running large language models on phones requires squeezing their weights into fewer bits, a process called quantization. Library writers must hand-craft a specialized kernel for every combination of compression format and hardware, leaving many useful combinations unsupported.
This work extends the Halide programming system to treat compression formats as reusable codecs (encoder-decoder pairs). A new approximate_by directive lets the system automatically schedule these codecs for any computation.
The result is hundreds of kernels from a single program, reaching 95 percent of hand-written speed in 91 percent of cases. It needs less code and even caught a bug in a popular quantization format.
