The paper introduces CanonQ, a framework for LLMs that compresses weights, activations, and KV caches together down to 2 bits. The key idea is to first transform each tensor source into a canonical form using fixed rotations and energy normalization, then reuse a single frozen codebook across all layers and models. Joint training then adapts the network to work well with the resulting quantization.
At the extreme setting of W2A4KV2, CanonQ-Omni achieved up to 14.28x lower perplexity on WikiText-2 and up to 57.9% higher average zero-shot accuracy compared to prior methods across LLaMA3 models (1B, 3B, 8B). Benefits also transferred to other models like Qwen3-1.7B and showed improvements in code generation (HumanEval +41.7%) and math reasoning (GSM8K +39.1%) at W2A16KV16 on MobileLLM-Pro-1B.