The paper presents moesq, a framework that compresses expert weights in Mixture-of-Experts (MoE) models into low-precision sparse representations that are native to hardware Sparse Tensor Cores (SpTCs). It uses a differentiable continuous reparameterization to jointly optimize sparsity masks and quantized weights under a router-weighted reconstruction objective, and develops a custom grouped sparse GEMM kernel for efficient execution on NVIDIA B200 GPUs.
On MoE models from 30 billion to one trillion parameters, moesq improves accuracy over prior joint sparse-quantization methods by up to 4.35 percentage points, retaining 96.09% of the original model's performance. The kernel achieves up to 1.65× speedup over dense baselines, leading to 1.18× higher serving throughput and up to 4.03× lower end-to-end latency. This addresses the memory and bandwidth bottleneck of trillion-parameter MoEs by exploiting previously underutilized hardware capabilities, making large-scale deployment more practical.