Search

9 results for “MMLU”

1. Research · Evaluation & benchmarks

   Scaling curves can be certified with far fewer sampled answers

   Cheap to Draw, Expensive to Trust: Certifying Test-Time Scaling Curves

   A variance-aware audit provides simultaneous confidence bands across test-time budgets while reducing the number of generated answers.

   MMLU-Pro: 79133 generated answers (as reported)

   7 Oct 2026PMCC AI Lab, Peter Munk Cardiac Centre, University Health Network

2. Research · Evaluation & benchmarks

   Language models’ own confidence becomes harder to outperform as they improve

   Predictable performance is suboptimal, and what it means for model routing

   Zhang and Hardt connect prediction gains beyond a model’s own confidence to its room for improvement, then test the relationship across 16 language models.

   MMLU-Pro: 0.65 Pearson correlation between empirical suboptimality and (as reported)

   23 Oct 2026

3. Research · Evaluation & benchmarks

   RAG systems can fail catastrophically when knowledge bases contain semantic errors

   RAGStress: A controlled benchmark for evaluating retrieval-augmented generation under knowledge-base degradation

   A new benchmark systematically tests how five language models handle four types of document corruption at three severity levels.

   RAGStress: 98.6% accuracy (as reported)

   6 Oct 2026Amazon

4. Research · Agents

   Revision-aware agent graphs reuse valid work while avoiding stale answers

   Revision-Aware Independent Agent Graphs for Dynamic Reasoning

   RIAG combines deterministic version selection, version-keyed caching and bounded independent reasoning to handle tasks revised over time.

   Full collection (6 benchmarks): 54.24% joint accuracy (as reported)

   2 Oct 2026Harvard AI and Robotics Lab, Harvard University

5. Research · Language models

   AI chatbots fail long tasks due to competing content, not length

   Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval

   Neutral filler costs nothing in retrieval tests, while a few rival statements crash accuracy and attention on the evidence predicts the loss.

   MMLU: 40.5% accuracy (as reported)

   4 Nov 2026University of California San DiegoCode

6. Research · Safety & security

   Local evaluation reveals activation patching can cause benign regressions even when global metrics hold steady

   PatchBench: Measuring Collateral Damage in Activation Patching

   PatchBench curates model-specific jailbreak failures and introduces a protocol to measure harmful correction and benign preservation around each repair.

   MMLU: 57.88% accuracy (as reported)

   9 Oct 2026LIX (École Polytechnique, IP Paris, CNRS), AMIAD (Agence Ministérielle pour l’IA de Défense)

7. Research · Evaluation & benchmarks

   Current date in system prompts skews LLM evaluation results

   Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation

   The authors show that LLM accuracy varies by up to 14% on math reasoning and reshuffles leaderboards when only the date changes, making the effect larger than other known sources of non-determinism.

   MMLU: 2.3% worst-to-best date delta in accuracy (as reported)

   6 Oct 2026Johannes Gutenberg University Mainz, Universidad Iberoamericana, University of Colorado Boulder

8. Research · Safety & security

   Language models favor suggested answers even when told the source is unreliable

   How Much Do You Trust A Dog? Quantifying Hint-Following in Reasoning Language Models

   Across three model families, the authors find structured shifts toward suggested answers, even when prompts explicitly warn that the source is unreliable.

   MMLU: 1.72 mean log-odds shift (as reported)

   26 Oct 2026Northwestern University, NSF-Simons AI Institute for the Sky (SkAI), Stanford University

9. Research · Evaluation & benchmarks

   Accessibility profiles reveal hidden tradeoffs in LLM reliability

   Revealing what benchmark scores hide: Jointly measuring the breadth and reliability of LLM capabilities using accessibility profiles

   Kaplan et al. show that models with similar average accuracy can differ sharply in how widely and dependably they succeed across equivalent wordings.

   MMLU: 10 pts Reliable@5 difference (LLaMA-3 8B minus LLaMA-3.1 8B) (as reported)

   20 Oct 2026University of Maryland, College Park