non-determinism

The papers

Most central to this idea first, not most recent.

• Independentcs.LG

  AI model rankings can reverse when prompts include solved examples

  Model-derived measurements should be corrected only when their repeatability is clear, and model rankings should be compared separately across prompt settings.

  Oct 2026

• Independentcs.CL

  Language models’ jailbreak weaknesses depend more on training randomness than data

  A model's vulnerability to specific jailbreaks is determined mostly by its pretraining run, not by its pretraining data or safety post-training seed.

  Oct 2026

• Top Universitycs.LG

  Mean benchmark accuracy misleads model release decisions; recipe cards certify tail risk

  Mean accuracy is the wrong estimand for model release decisions; tail failure probability should be certified instead.

  Seoul National University Hospital, Inha University · Oct 2026

• Big Techcs.CL

  Text classifier predictions shift with serving context, even with fixed model and input

  Text classifier predictions change with padding length, batch shape, precision, backend, and other serving settings even when inputs and model weights are fixed.

  Amazon · Oct 2026

• Independentcs.CL

  LLM judges give inconsistent verdicts even at zero temperature

  LLM judges produce different verdicts for identical inputs at temperature zero due to cloud infrastructure.

  Oct 2026

• Research Labcs.AIcode

  Agent performance depends more on system configuration than model size

  Agent performance depends more on system configuration, especially supplied information, than on model choice alone, and reliable verification requires system tools rather than prompts.

  Technical University of Munich, MCML · Oct 2026

• Academiccs.CL

  Current date in system prompts skews LLM evaluation results

  Hidden dates in system prompts cause LLM evaluation scores to fluctuate by up to 14% and reorder model rankings.

  Johannes Gutenberg University Mainz, Universidad Iberoamericana · Oct 2026

• Top Universitycs.AI

  LLM judges as measurement instruments fail reliability tests on shared endpoints

  Black-box LLM judges on shared endpoints are unreliable measurement instruments due to unstable rankings and biased readouts.

  University of Sheffield, Ranplan Wireless Network Design Ltd. · Sept 2026

• Industrycs.CR

  Tampered embeddings hijack offline AI retrieval despite tolerance checks

  Exact verification of quantisation codes closes the text-vector integrity gap in offline RAG packs.

  securagen.ai · Nov 2026

• Top Universitycs.CL

  LLM agents decide to follow prompt injections during reasoning, not at injection time

  For reasoning LLM agents, the decision to follow a prompt injection is made during reasoning, not when the injection is read.

  Fullive-AI, Nanyang Technological University · Oct 2026

• Academiccs.CR

  Signed LLM weights do not certify the quantized files users actually run

  Signing full-precision LLM weights does not verify deployed quantized models unless the transformation inputs and build environment support bit-exact reproduction.

  National Yang Ming Chiao Tung University, National Taiwan University of Science and Technology · Oct 2026

• Academiccs.CL

  LLM judge scores can stay stable while individual verdicts shift substantially

  LLM judge verdicts are unstable: rerunning the same Arena-Hard setup changes 21.2% of prompt outcomes even though aggregate scores move only about one point.

  University College London · Oct 2026

• Big Techcs.HC

  AI agents can repeat tasks consistently when successful runs become code

  FlashScript compiles successful agent runs into neuro-symbolic programs with identical tool calls, lower token cost, and reuse on unseen instances.

  University of Southern California, Microsoft · Oct 2026

• Academiccs.LG

  Jailbreak success rates flip with evaluator and sampling choices

  Jailbreak attack success rates are not stable properties: evaluation method, hyperparameters, and sampling budget can reverse rankings and hide unsafe responses.

  University of Oldenburg, Chalmers University of Technology · Oct 2026

• Industry

  Small-Sample LLM Rankings Often Overstate Evaluation Reproducibility

  LLM evaluation rankings often look definitive but are highly unstable under resampling and defensible analysis changes.

  Skelf Research · Sept 2026

• Top Universitycs.AIcode

  More Harness Programs Do Not Necessarily Produce Useful Specialization

  Repeatable score gains from different LLM harness programs are dominated by persistent weaknesses, not useful specialization.

  The Chinese University of Hong Kong, New Laboratory of Pattern Recognition (NLPR) · Sept 2026

• Big Techcs.PL

  Three-step hybrid pipeline validates GPU kernels by combining concurrency testing with formal proof

  RESOLVE validates GPU kernels by testing for nondeterminism, reducing concurrency, and proving functional equivalence.

  University of California, Riverside, Math, Inc. · Oct 2026

• Academiccs.AI

  Prompt optimization yields no gains on strong reasoning models

  Automated prompt optimization produces no statistically significant accuracy gains on frontier reasoning models; it fixes formatting errors but cannot supply missing knowledge or fit truncated context.

  Hunan Agricultural University · Nov 2026

• Top Universitycs.LG

  Language models can lag 50 to 150 updates behind training

  Standard bf16 rounding silently discards most RL updates, creating a hidden lag that unbiased rounding eliminates.

  Stanford University · Nov 2026

• Industrycs.LG

  AI services secretly using cheaper models can be detected with few queries

  API-TAIL detects partial model substitution in text-only language-model services by adjusting for legitimate serving changes and counting persistently unusual responses.

  TokenRhythm Technologies · Oct 2026

• Big Techcs.AI

  Optimizing AI agents for cost as well as quality improves both

  Separating candidate promotion from final deployment choice helps agent-harness optimization preserve useful quality-cost trade-offs and produce better, cheaper solutions.

  University College London, Pohang University of Science and Technology · Nov 2026

• Industrycs.CL

  LLM judges can be consistent yet wrong and order-sensitive

  LLM judge verdicts are not reliable: they vary across identical runs, flip with presentation order, and determinism can hide near-chance accuracy.

  PayPal Artificial Intelligence, PayPal · Oct 2026

• Academiccs.AI

  Coding harnesses change token costs more reliably than success rates

  Swapping a coding agent's harness moves pass rate no more than rerunning the same harness.

  Shandong University · Oct 2026

• Big Techcs.LG

  Reinforcement learning makes models’ admissions of failure hard to reproduce

  Failure disclosure in language models varies widely across reinforcement learning training runs, even when task accuracy remains stable.

  Stanford University, Anthropic · Sept 2026

• Industrycs.LG

  Fixed controls expose limits in financial LLM agent evaluation

  Fixed-response replay and matched defect controls show execution comparisons confound model behavior, while target recall alone misrepresents audit quality.

  Virginia Tech · Sept 2026

• Big Techcs.CLcode

  Cut-point replay turns recorded agent failures into regression tests

  Cut-point replay converts recorded LLM agent failures into deterministic regression tests that detect faulty code changes.

  Microsoft · Sept 2026

• Industrycs.CL

  Swapping the AI judge often changes which chatbot ranks best

  Swapping in a judge model from another provider does not preserve system rankings, and judges agreeing flags a bad judge but does not certify a conclusion.

  CEO Business School · Nov 2026

• Big Techcs.CV

  Video benchmark scores shift sharply with hidden evaluation choices

  Unreported evaluation choices can shift video benchmark scores by up to 19.5 percentage points, and the curated Meta-Video benchmark preserves rankings with 5x less compute.

  University of Tübingen, Max Planck Institute for Informatics, SIC · Oct 2026

• Top Universitycs.CR

  Unverified state text can flip calibrated model decisions

  Appending an unverified opinion to the state flips 12.1% of decisions in RLCD models, making the state an untrusted input.

  Institute of Information Engineering, Chinese Academy of Sciences, University of Chinese Academy of Sciences · Sept 2026

• Top Universitycs.SE

  Only 5.3% of LLM call sites in agentic systems enforce all expected constraints

  Only 5.3% of LLM call sites enforce their full contract, leaving 63% of expected constraints unmet.

  Nanjing University, Xi'an Jiaotong University · Oct 2026

• Big Techcs.AI

  Memory guidelines help language-model agents succeed consistently across repeated runs

  LLM agents averaging 77% per-run success on AppWorld pass all five runs on only 53% of tasks; self-evolving consistency guidelines raise this by 16 points same-task and 13 similar-task.

  IBM Software Innovation Lab, IBM Research · Sept 2026