Search

35 results for “MMLU” · page 1 of 2

1. Research · Agents

   Draft-KV ensures the receiver actually uses the sender's latent message

   Draft-KV: Learning Useful Latent Communication Between Language Models

   By sending key-value states from the sharer's draft, a lightweight interface boosts a 0.5B receiver to 78% on MMLU-Redux, and message-replacement tests confirm the gain comes from genuine information use.

   MMLU-Redux: 46.11% accuracy (as reported)

   29 Sept 2026City University of Hong Kong, University of Science and Technology of China, University of Electronic Science and Technology of China

2. Research · Evaluation & benchmarks

   Scaling curves can be certified with far fewer sampled answers

   Cheap to Draw, Expensive to Trust: Certifying Test-Time Scaling Curves

   A variance-aware audit provides simultaneous confidence bands across test-time budgets while reducing the number of generated answers.

   MMLU-Pro: 192000 generated answers (as reported)

   7 Oct 2026PMCC AI Lab, Peter Munk Cardiac Centre, University Health Network

3. Research · Language models

   Query-adaptive neuron steering improves frozen language models across benchmarks

   Query-Adaptive Neuron Steering for LLMs

   Q-NeuS uses each query’s internal activation pattern to select and scale neuron interventions without updating the underlying model.

   MMLU-Pro: 59.11% accuracy (as reported)

   15 Oct 2026Northeastern University, Washington University in St. Louis, Nanjing University of Aeronautics and Astronautics

4. Research · Reinforcement learning

   Turning failed reasoning attempts into training signal improves GRPO on hard problems

   Failure Is the Mother of Success: Recovering Supervision with Grounding Hindsight

   RSHD generates corrective notes from failures, verifies them via replays, and distills successful behaviors into the policy without extra inference cost.

   MMLU-Pro: 61.2% accuracy (as reported)

   16 Oct 2026University of California, San Diego, Adobe Research, University of Virginia

5. Research · Agents

   Progressive pruning lets language-model debates beat equal-cost baselines

   Beyond Solo and Consistency: Vindicating Multi-Agent Debate via Conditional Progressive Pruning

   Conditional Progressive Pruning uses intermediate debate results and learned agent behavior to allocate computation across rounds more effectively.

   MMLU-Pro: 66.33% accuracy (as reported)

   5 Oct 2026Rutgers University, New Brunswick, University of Cambridge, Tsinghua University

6. Research · Efficiency & systems

   Multi-precision LLM weight format keeps a single prefix-readable stream

   StagQ: Constraint-Driven Multi-Precision Weight Quantization for LLMs

   StagQ stores weights as a 2-bit base with refinement planes, using a side record for outliers, and outperforms baselines on MMLU at 2 and 3 bits.

   MMLU: 38.38% accuracy (as reported)

   8 Oct 2026Huawei Technologies Ltd.

7. Research · Evaluation & benchmarks

   Language models’ own confidence becomes harder to outperform as they improve

   Predictable performance is suboptimal, and what it means for model routing

   Zhang and Hardt connect prediction gains beyond a model’s own confidence to its room for improvement, then test the relationship across 16 language models.

   MMLU-Pro: 0.82 Pearson correlation between empirical suboptimality and (as reported)

   23 Oct 2026

8. Research · Evaluation & benchmarks

   RAG systems can fail catastrophically when knowledge bases contain semantic errors

   RAGStress: A controlled benchmark for evaluating retrieval-augmented generation under knowledge-base degradation

   A new benchmark systematically tests how five language models handle four types of document corruption at three severity levels.

   RAGStress: 41.1% accuracy (as reported)

   6 Oct 2026Amazon

9. Research · Language models

   Role prompts help language models only under the right conditions

   Cognitive Enhancement: Rethinking the Necessity of Role-Playing for Large Language Models

   Across multilingual knowledge benchmarks, persona prompting varied with model capacity, subject and language, while combining equivalent prompts across languages produced more consistent results.

   MMLU: 0.88 accuracy (as reported)

   7 Oct 2026Xiamen University, Tongyi Lab, The Chinese University of Hong Kong

10. Research · Agents

    Revision-aware agent graphs reuse valid work while avoiding stale answers

    Revision-Aware Independent Agent Graphs for Dynamic Reasoning

    RIAG combines deterministic version selection, version-keyed caching and bounded independent reasoning to handle tasks revised over time.

    Full collection (6 benchmarks): 54.24% joint accuracy (as reported)

    2 Oct 2026Harvard AI and Robotics Lab, Harvard University

11. Research · Language models

    Alternating context edits and logit steering boosts frozen LLM accuracy

    EMO: A Closed-Loop Test-Time Computation Framework via Alternating Context Optimization and Logits Steering

    EMO closes the loop between feedback-driven context revision and decoding-time distribution correction, improving zero-shot performance across eight models and six benchmarks.

    MMLU-Pro: 66.6% accuracy (as reported)

    16 Oct 2026Peking University, Samsung Research

12. Research · Reinforcement learning

    Posterior rewards become indistinguishable for long reasoning traces in LLM training

    Rethinking Probability-Based Reinforcement Learning From Posterior Concentration

    The authors identify a length-dependent failure mode they call Posterior Concentration Phenomenon, and propose a framework to mitigate it.

    MMLU-Pro: 61.3% Avg@2 (as reported)

    7 Oct 2026National University of Singapore, The Hong Kong University of Science and Technology, University of Science and Technology of China

13. Research · Agents

    Joint harness search balances model accuracy, safety, and token use

    MoMHa: Multi-Objective Optimization of LLM Harnesses over Accuracy, Safety, and Tokens

    Across 17 domains and 12 models, jointly optimizing the code around language models outperformed sequential and accuracy-only approaches.

    3D hypervolume: 0.48 hypervolume indicator (as reported)

    28 Sept 2026Adobe Research

14. Research · Language models

    AI chatbots fail long tasks due to competing content, not length

    Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval

    Neutral filler costs nothing in retrieval tests, while a few rival statements crash accuracy and attention on the evidence predicts the loss.

    MMLU: 51.7% accuracy (as reported)

    4 Nov 2026University of California San DiegoCode

15. Research digest

    This Week in Agentic AI Research: Runtime Contracts Beat Model Intuition

    The strongest results move reliability from model reasoning into explicit state, memory, communication and audit mechanisms.

    1 Oct 2026

16. Research · Efficiency & systems

    Tetra serves 2.7-bit language models without expanding lattice codes

    Tetra: Serving Leech-Lattice Quantized LLMs at 2.7 Bits per Parameter

    A fused GPU kernel decodes compact Leech-lattice weights during multiplication, reducing memory traffic while retaining usable accuracy on Qwen3 models.

    MMLU: 63.37% accuracy (as reported)

    5 Oct 2026

17. Research · Safety & security

    Method to permanently erase unwanted LLM personas via weight edits

    LLM Persona Unlearning

    A new benchmark and contrastive erasure technique suppress target personas while preserving utility across six models and five personas.

    PersonaUnlearnBench: 100% TF (as reported)

    1 Oct 2026University of Macau, Mila – Québec AI Institute, McGill University

18. Research · Efficiency & systems

    Combining model compression techniques can collapse small language models

    Compression Technique Interactions in Small Language Models: A Factorial Ablation Study

    A systematic study of 72 compression configurations shows that INT8 quantization is lossless, while NF4 with pruning causes total failure, and LoRA fine-tuning recovers most quality without a teacher model.

    WikiText-2: 34.74 perplexity (as reported)

    3 Nov 2026Amazon Web Services

19. Research · Safety & security

    Local evaluation reveals activation patching can cause benign regressions even when global metrics hold steady

    PatchBench: Measuring Collateral Damage in Activation Patching

    PatchBench curates model-specific jailbreak failures and introduces a protocol to measure harmful correction and benign preservation around each repair.

    MMLU: 63.05% accuracy (as reported)

    9 Oct 2026LIX (École Polytechnique, IP Paris, CNRS), AMIAD (Agence Ministérielle pour l’IA de Défense)

20. Research · Evaluation & benchmarks

    Current date in system prompts skews LLM evaluation results

    Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation

    The authors show that LLM accuracy varies by up to 14% on math reasoning and reshuffles leaderboards when only the date changes, making the effect larger than other known sources of non-determinism.

    MMLU: 2.27% worst-to-best date delta in accuracy (as reported)

    6 Oct 2026Johannes Gutenberg University Mainz, Universidad Iberoamericana, University of Colorado Boulder