News without the noise
Benchmark
Reported in 43 papers on Zotpaper. The latest, each as the paper states it:
22 results for “MMLU” · page 1 of 2
Research · Evaluation & benchmarks
Cheap to Draw, Expensive to Trust: Certifying Test-Time Scaling Curves
A variance-aware audit provides simultaneous confidence bands across test-time budgets while reducing the number of generated answers.
MMLU-Pro: 0.74 accuracy (as reported)
PMCC AI Lab, Peter Munk Cardiac Centre, University Health Network
Research · Language models
Query-Adaptive Neuron Steering for LLMs
Q-NeuS uses each query’s internal activation pattern to select and scale neuron interventions without updating the underlying model.
MMLU-Pro: 59.11% accuracy (as reported)
Northeastern University, Washington University in St. Louis, Nanjing University of Aeronautics and Astronautics
Research · Reinforcement learning
Failure Is the Mother of Success: Recovering Supervision with Grounding Hindsight
RSHD generates corrective notes from failures, verifies them via replays, and distills successful behaviors into the policy without extra inference cost.
MMLU-Pro: 40.4% accuracy (as reported)
University of California, San Diego, Adobe Research, University of Virginia
Research · Efficiency & systems
StagQ: Constraint-Driven Multi-Precision Weight Quantization for LLMs
StagQ stores weights as a 2-bit base with refinement planes, using a side record for outliers, and outperforms baselines on MMLU at 2 and 3 bits.
MMLU: 38.38% accuracy (as reported)
Huawei Technologies Ltd.
Research · Evaluation & benchmarks
Predictable performance is suboptimal, and what it means for model routing
Zhang and Hardt connect prediction gains beyond a model’s own confidence to its room for improvement, then test the relationship across 16 language models.
MMLU-Pro: 0.65 Pearson correlation between empirical suboptimality and (as reported)
Research · Evaluation & benchmarks
RAGStress: A controlled benchmark for evaluating retrieval-augmented generation under knowledge-base degradation
A new benchmark systematically tests how five language models handle four types of document corruption at three severity levels.
RAGStress: 86.7% accuracy (as reported)
Amazon
Research · Language models
Cognitive Enhancement: Rethinking the Necessity of Role-Playing for Large Language Models
Across multilingual knowledge benchmarks, persona prompting varied with model capacity, subject and language, while combining equivalent prompts across languages produced more consistent results.
MMLU: 0.59 accuracy (as reported)
Xiamen University, Tongyi Lab, The Chinese University of Hong Kong
Research · Language models
EMO: A Closed-Loop Test-Time Computation Framework via Alternating Context Optimization and Logits Steering
EMO closes the loop between feedback-driven context revision and decoding-time distribution correction, improving zero-shot performance across eight models and six benchmarks.
MMLU-Pro: 70.9% accuracy (as reported)
Peking University, Samsung Research
Research · Reinforcement learning
Rethinking Probability-Based Reinforcement Learning From Posterior Concentration
The authors identify a length-dependent failure mode they call Posterior Concentration Phenomenon, and propose a framework to mitigate it.
MMLU-Pro: 61.3% Avg@2 (as reported)
National University of Singapore, The Hong Kong University of Science and Technology, University of Science and Technology of China
Research · Language models
Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval
Neutral filler costs nothing in retrieval tests, while a few rival statements crash accuracy and attention on the evidence predicts the loss.
MMLU: 51.7% accuracy (as reported)
University of California San DiegoCode
Research · Efficiency & systems
Compression Technique Interactions in Small Language Models: A Factorial Ablation Study
A systematic study of 72 compression configurations shows that INT8 quantization is lossless, while NF4 with pruning causes total failure, and LoRA fine-tuning recovers most quality without a teacher model.
WikiText-2: 920.56 perplexity (as reported)
Amazon Web Services
Research · Safety & security
PatchBench: Measuring Collateral Damage in Activation Patching
PatchBench curates model-specific jailbreak failures and introduces a protocol to measure harmful correction and benign preservation around each repair.
MMLU: 57.88% accuracy (as reported)
LIX (École Polytechnique, IP Paris, CNRS), AMIAD (Agence Ministérielle pour l’IA de Défense)
Research · Safety & security
How Much Do You Trust A Dog? Quantifying Hint-Following in Reasoning Language Models
Across three model families, the authors find structured shifts toward suggested answers, even when prompts explicitly warn that the source is unreliable.
MMLU: 73.2% target rate (as reported)
Northwestern University, NSF-Simons AI Institute for the Sky (SkAI), Stanford University
Research · Language models
On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models
Sick and colleagues report gains in language and visual reasoning using self-generated supervision rather than human-curated training questions or reference answers.
MMLU: 47.7% accuracy (as reported)
Google, Ulm University
Research · Language models
Head Dimension Matters in Language Models for Long-Context Retrieval
Under a fixed parameter budget, models with fewer but wider heads outperform those with narrower heads on retrieval tasks.
RULER-100: 31.25% overall score (as reported)
Preferred Networks, Inc.
Research · Evaluation & benchmarks
Revealing what benchmark scores hide: Jointly measuring the breadth and reliability of LLM capabilities using accessibility profiles
Kaplan et al. show that models with similar average accuracy can differ sharply in how widely and dependably they succeed across equivalent wordings.
MMLU: -9.3 pts Access@5 difference (LLaMA-3 8B minus LLaMA-3.1 8B) (as reported)
University of Maryland, College Park
Research · Efficiency & systems
SQuAT: Spherical Lattice Vector Quantization-Aware Training
Loeschcke et al. decompose weight blocks into direction and magnitude and apply factored straight-through estimators, improving accuracy over joint-STE baselines and matching scalar INT-QAT below 2.5 bits per weight.
MMLU: 69.5% accuracy (as reported)
University of Copenhagen, Qualcomm AI Research
Research · Safety & security
SafeEvo: Deciphering the Safety Alignment Mechanism and Evolution in Language Models
SafeEvo identifies sparse refusal circuits in base models, traces their structural drift across safety checkpoints, and uses them to improve fine-tuning.
HarmBench: 6.47 pts average ASR reduction (as reported)
The University of Hong Kong, Chinese Academy of Sciences, Information Engineering University
Research · Efficiency & systems
Deflating the Hessian: Rank-4 W4A4 Quantization for Multimodal Diffusion Transformers
A unified framework jointly modeling weight and activation quantization error achieves better performance with rank 4 than existing methods with rank 32.
MMLU: 68.17% accuracy (as reported)
The University of Hong Kong, Huawei Technologies Co., Ltd.
Research · Language models
MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface
MetaEncoder uses one natural-language interface to score options in closed sets and retrieve items from collections containing millions of candidates.
MMLU: 0.78 Hit@1 (as reported)
Meta AI