Search

6 results for “GSM8K”

1. Research · Evaluation & benchmarks

   Models flag missing facts more readily than usable unsupported premises

   Models Verify Sufficiency, Not Validity: Usable Falsehoods Evade the Checks that Gaps Trigger

   Controlled tests found that making an unsupported premise usable reduced models’ warnings, although accepting question-supplied premises can also reflect conditional reasoning.

   GSM8K: 0.91 flagged unanswerable (as reported)

   27 Oct 2026Google

2. Research · Language models

   Prompt optimization yields no gains on strong reasoning models

   Compile as Hypothesis: A Headroom Typology for Prompt Optimization on Real Reasoning Models

   Automated prompt compilers show no statistically significant accuracy improvements across four optimizer families and eight tasks, but structured genomes prevent catastrophic search failures.

   AIME 2024+25: 60 problems correct (as reported)

   3 Nov 2026Hunan Agricultural University

3. Research · Evaluation & benchmarks

   Current date in system prompts skews LLM evaluation results

   Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation

   The authors show that LLM accuracy varies by up to 14% on math reasoning and reshuffles leaderboards when only the date changes, making the effect larger than other known sources of non-determinism.

   GSM8K: 7.75% worst-to-best date delta in accuracy (as reported)

   6 Oct 2026Johannes Gutenberg University Mainz, Universidad Iberoamericana, University of Colorado Boulder

4. Research · Evaluation & benchmarks

   Pass@k misses output diversity changes caused by model post-training

   What pass@k Cannot Measure: Evaluating Diversity and Capability Retention after Post-Training

   GRPO and rejection-sampling fine-tuning produced sharply different output distributions despite often similar pass@k scores.

   GSM8K: 0.51 distinct-3 (verifier-correct completions) (as reported)

   8 Oct 2026Accenture Strategy & Consulting, Vizuara AI Labs

5. Research · Evaluation & benchmarks

   Reused evaluation sets exaggerate gains in LLM self-improvement loops

   The Winner's Curse in LLM Self-Improvement Loops: Selection Noise, Lock-in, and Acceptance Rules

   Experiments with prompt-rewriting systems show that repeatedly selecting candidates on small evaluation sets creates large gaps between reported and held-out performance.

   GSM8K: 19.6 pts proxy-held-out gap (as reported)

   8 Oct 2026Meta, Microsoft

6. Research · Evaluation & benchmarks

   Language models’ own confidence becomes harder to outperform as they improve

   Predictable performance is suboptimal, and what it means for model routing

   Zhang and Hardt connect prediction gains beyond a model’s own confidence to its room for improvement, then test the relationship across 16 language models.

   GSM8K: 0.91 Pearson correlation between empirical suboptimality and (as reported)

   23 Oct 2026