News without the noise
Benchmark
Reported in 43 papers on Zotpaper. The latest, each as the paper states it:
11 results for “MMLU”
Research · Efficiency & systems
StagQ: Constraint-Driven Multi-Precision Weight Quantization for LLMs
StagQ stores weights as a 2-bit base with refinement planes, using a side record for outliers, and outperforms baselines on MMLU at 2 and 3 bits.
MMLU: 38.38% accuracy (as reported)
Huawei Technologies Ltd.
Research · Evaluation & benchmarks
Predictable performance is suboptimal, and what it means for model routing
Zhang and Hardt connect prediction gains beyond a model’s own confidence to its room for improvement, then test the relationship across 16 language models.
MMLU-Pro: 0.82 Pearson correlation between empirical suboptimality and (as reported)
Research · Agents
MoMHa: Multi-Objective Optimization of LLM Harnesses over Accuracy, Safety, and Tokens
Across 17 domains and 12 models, jointly optimizing the code around language models outperformed sequential and accuracy-only approaches.
3D hypervolume: 0.48 hypervolume indicator (as reported)
Adobe Research
Research · Efficiency & systems
Tetra: Serving Leech-Lattice Quantized LLMs at 2.7 Bits per Parameter
A fused GPU kernel decodes compact Leech-lattice weights during multiplication, reducing memory traffic while retaining usable accuracy on Qwen3 models.
MMLU: 69.58% accuracy (as reported)
Research · Efficiency & systems
Compression Technique Interactions in Small Language Models: A Factorial Ablation Study
A systematic study of 72 compression configurations shows that INT8 quantization is lossless, while NF4 with pruning causes total failure, and LoRA fine-tuning recovers most quality without a teacher model.
WikiText-2: 0.99 Pearson correlation between 20-sample and full-corpus (as reported)
Amazon Web Services
Research · Efficiency & systems
SQuAT: Spherical Lattice Vector Quantization-Aware Training
Loeschcke et al. decompose weight blocks into direction and magnitude and apply factored straight-through estimators, improving accuracy over joint-STE baselines and matching scalar INT-QAT below 2.5 bits per weight.
MMLU: 69.5% accuracy (as reported)
University of Copenhagen, Qualcomm AI Research
Research · Efficiency & systems
Deflating the Hessian: Rank-4 W4A4 Quantization for Multimodal Diffusion Transformers
A unified framework jointly modeling weight and activation quantization error achieves better performance with rank 4 than existing methods with rank 32.
MMLU: 68.17% accuracy (as reported)
The University of Hong Kong, Huawei Technologies Co., Ltd.
Research · Efficiency & systems
Tailoring the Quantization Space for 1-Bit KV Cache Compression
TaSQ uses query-guided weighting, cross-head normalization, and covariance-aware grouping to improve VQ quality at extreme compression rates.
MMLU: 71.26% accuracy (as reported)
Seoul National University, Stanford University
Research · Efficiency & systems
DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving
DuoRate stores weights once and adapts read precision per workload, matching 4-bit accuracy at 2.9 bpw
MMLU: 80.56% accuracy (as reported)
Huawei Technologies Ltd.
Research · Efficiency & systems
Quantization Error Is Spectrally Flat: A Single Random Probe Is a Calibrated, Data-Free Sensitivity Estimator, with Application to Budget-Targeted Mixed-Precision Quantization
The authors show that spectrally flat rounding errors make Gaussian probes reliable enough to allocate per-tensor bit widths under an exact memory budget.
MMLU: 73.1% accuracy (as reported)
Blacksheep ai.
Research · Efficiency & systems
Delta-Matching: Closing the Final Gap of Native 8-bit Training for LLMs
The method corrects a forward-backward mismatch in quantized softmax attention, matching mixed-precision training across tested architectures and stages.
MMLU: 34% accuracy (as reported)
Carnegie Mellon University, NVIDIA, Massachusetts Institute of Technology