News without the noise
35 results for “MMLU” · page 2 of 2
Research · Safety & security
How Much Do You Trust A Dog? Quantifying Hint-Following in Reasoning Language Models
Across three model families, the authors find structured shifts toward suggested answers, even when prompts explicitly warn that the source is unreliable.
MMLU: 95.96% target rate (as reported)
Northwestern University, NSF-Simons AI Institute for the Sky (SkAI), Stanford University
Research · Language models
On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models
Sick and colleagues report gains in language and visual reasoning using self-generated supervision rather than human-curated training questions or reference answers.
MMLU: 47.7% accuracy (as reported)
Google, Ulm University
Research · Language models
Head Dimension Matters in Language Models for Long-Context Retrieval
Under a fixed parameter budget, models with fewer but wider heads outperform those with narrower heads on retrieval tasks.
RULER-100: 41.21% overall score (as reported)
Preferred Networks, Inc.
Research · Safety & security
TRACE: Trajectory Return Attribution and Contrastive Erasure for Multi-Turn Safety
A token-level safety objective outperformed alternatives across 35 model and attack combinations, with benchmark utility falling by no more than 1.23 points.
MMLU and HellaSwag: 1.23 pts accuracy drop (as reported)
PRADA Lab, King Abdullah University of Science and Technology, State Key Laboratory of Internet of Things for Smart City
Research · Evaluation & benchmarks
Revealing what benchmark scores hide: Jointly measuring the breadth and reliability of LLM capabilities using accessibility profiles
Kaplan et al. show that models with similar average accuracy can differ sharply in how widely and dependably they succeed across equivalent wordings.
MMLU: 37% median brittle share (as reported)
University of Maryland, College Park
Research · Efficiency & systems
SQuAT: Spherical Lattice Vector Quantization-Aware Training
Loeschcke et al. decompose weight blocks into direction and magnitude and apply factored straight-through estimators, improving accuracy over joint-STE baselines and matching scalar INT-QAT below 2.5 bits per weight.
MMLU: 69.5% accuracy (as reported)
University of Copenhagen, Qualcomm AI Research
Research · Safety & security
SafeEvo: Deciphering the Safety Alignment Mechanism and Evolution in Language Models
SafeEvo identifies sparse refusal circuits in base models, traces their structural drift across safety checkpoints, and uses them to improve fine-tuning.
HarmBench: 5.35% ASR (as reported)
The University of Hong Kong, Chinese Academy of Sciences, Information Engineering University
Research · Efficiency & systems
Deflating the Hessian: Rank-4 W4A4 Quantization for Multimodal Diffusion Transformers
A unified framework jointly modeling weight and activation quantization error achieves better performance with rank 4 than existing methods with rank 32.
MMLU: 68.17% accuracy (as reported)
The University of Hong Kong, Huawei Technologies Co., Ltd.
Research · Language models
MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface
MetaEncoder uses one natural-language interface to score options in closed sets and retrieve items from collections containing millions of candidates.
MMLU: 0.78 Hit@1 (as reported)
Meta AI
Research · Efficiency & systems
Tailoring the Quantization Space for 1-Bit KV Cache Compression
TaSQ uses query-guided weighting, cross-head normalization, and covariance-aware grouping to improve VQ quality at extreme compression rates.
MMLU: 71.26% accuracy (as reported)
Seoul National University, Stanford University
Research · Efficiency & systems
DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving
DuoRate stores weights once and adapts read precision per workload, matching 4-bit accuracy at 2.9 bpw
MMLU: 80% accuracy (as reported)
Huawei Technologies Ltd.
Research · Efficiency & systems
Quantization Error Is Spectrally Flat: A Single Random Probe Is a Calibrated, Data-Free Sensitivity Estimator, with Application to Budget-Targeted Mixed-Precision Quantization
The authors show that spectrally flat rounding errors make Gaussian probes reliable enough to allocate per-tensor bit widths under an exact memory budget.
MMLU: 73.1% accuracy (as reported)
Blacksheep ai.
Research · Efficiency & systems
Delta-Matching: Closing the Final Gap of Native 8-bit Training for LLMs
The method corrects a forward-backward mismatch in quantized softmax attention, matching mixed-precision training across tested architectures and stages.
MMLU: 34% accuracy (as reported)
Carnegie Mellon University, NVIDIA, Massachusetts Institute of Technology
Research · Language models
Composable Decoding on the Probability Simplex: Theory and Implementation
The authors formalise decoding as a regularised optimisation problem, recovering standard methods as special cases and enabling new decoders through compositional regularisers.
GPQA Diamond: 26.3% pass@1 (as reported)
Huawei Noah's Ark Lab, UCL Centre for AI
Research · Language models
Function Over Form: Distributional Orthogonalization in Mixture-of-Experts with Replica Expert Mechanism
The authors introduce DO-loss to penalize overlapping token assignments and REM to rebalance workloads without sacrificing specialization.
MMLU: 37.18% accuracy (as reported)
Peking University, Tencent Hunyuan