Evaluation & benchmarks

New benchmarks, evaluation methodology, measuring what models can do

97 articles · page 2 of 3

All Agents Language models Reinforcement learning Vision Generative media Speech & audio Robotics & embodied AI Safety & security Evaluation & benchmarks Efficiency & systems Code & software AI for science & health Theory

Viewgrid list compact

Formal audit detects omitted source relationships in generative search

The authors specify a claim-gated audit procedure for generative search answers that checks whether a source relationship omitted from the answer is resolved. The procedure requires that relationship evidence, answer adoption, materiality, and disclosure are all observed; incomplete evidence remains unresolved. On an exhaustive synthetic suite, the reference checker reproduces all 81 three-state predicate combinations and rejects 192 deliberately malformed records.

29 Sept 2026

Evaluation & benchmarks

Coding skill does not guarantee effective multi-agent software collaboration

The authors introduce a benchmark for evaluating collaboration among asynchronous coding agents using executable checks for dependencies between their assigned components. Across model families, scales, generations, and coordination protocols, stronger coding performance did not consistently produce better dependency resolution, while ordinary task metrics sometimes concealed coordination failures.

29 Sept 2026

Evaluation & benchmarks

Current LLM agents struggle to sustain long-horizon team collaboration

The authors introduce AgentWorld, an MMORPG-based benchmark designed to isolate collaboration among independently acting LLM agents with asymmetric roles and abilities. Across four model families, even the strongest system succeeded on only 52.0 percent of tasks, with failures linked to broken communication, role confusion and lost shared plans.

29 Sept 2026

Evaluation & benchmarks

False accusations can make LLM agents sabotage correct work

The authors tested 14 models on 365 tasks where an agent encountered an unsupported accusation after reaching a verified correct state. In Claude Code, false accusations damaged correct work in as many as 60.06% of runs, while a harness-level intervention reduced replayed harm by 74%.

29 Sept 2026

Evaluation & benchmarks

Agent skill for plugin migration shows reward gains but hides grading errors

The authors studied 64 reports on 16 static migration tasks, with two attempts per condition and 328 criterion decisions, evaluating a shipped plugin-upgrade skill. They found that the skill raised mean recorded reward by 4.92 points, but the gain was concentrated in one task, and eight task pairs were at the ceiling. Contract inspection and executable probes exposed grading errors that favored either arm, and re-grading with alternative judges gave gains of 10.63 and 6.09 points.

29 Sept 2026

Evaluation & benchmarks

Compiler turns scientific protocols into certified simulated laboratory tasks

Qin et al. present an agentic pipeline for converting scientific protocols into executable, verifiable tasks for embodied laboratory agents. They use it to build SciVLABench, a collection of 300 certified tasks, where the strongest tested policy achieves only 49.7 percent average success.

29 Sept 2026

Evaluation & benchmarks

Web agents struggle to turn search results into usable artifacts

Gill and colleagues introduce KNOWS, a benchmark of long, open-ended workflows spanning web research, information synthesis, interface navigation, and artifact creation in Google Workspace. Frontier computer-use agents often completed parts of these workflows, but the best system fully succeeded on fewer than 3% of tasks.

29 Sept 2026

Evaluation & benchmarks

Retrieval-based checks miss many factual errors in open-ended medical answers

Huang et al. examine why retrieve-then-verify systems struggle to identify factual errors in long-form medical answers. Using an automatically induced error taxonomy, they find an order-of-magnitude performance collapse from closed-ended benchmarks to the clinician-annotated MedExpert dataset, with larger models, additional reasoning and broader evidence sources failing to resolve the problem.

29 Sept 2026

Evaluation & benchmarks

Small-Sample LLM Rankings Often Overstate Evaluation Reproducibility

Sarkar repeatedly tested eight open model variants on prompt-structure inference, preserving 293 intermediate representations and auditing both output consistency and the resulting rankings. The models frequently produced different structures for identical calls, while bootstrap resampling and alternative merge rules showed that the evaluation identified the worst performers more reliably than the best.

29 Sept 2026

Evaluation & benchmarks

Structured evidence reports reduce false success after tool failures

Zhu et al. introduce Failure-Transparent Agents, a 100-task benchmark that tests how language models report outcomes after a required tool has already failed. Across six models and 3,600 human-annotated responses, a structured evidence contract reduced false-success and fabricated-detail rates to 0.8% each.

29 Sept 2026

Evaluation & benchmarks

Semantic alignment improves edge classification across unseen dynamic graph domains

Bonnet and Rei introduce a transfer-learning protocol for classifying future interactions in dynamic text-attributed graphs from previously unseen domains. Their experiments show that evaluated self-supervised graph methods underperform a simple text-only baseline on average, while their Spatio-Temporal Semantic Alignment model surpasses both that baseline and the existing methods.

29 Sept 2026

Evaluation & benchmarks

Goal pressure exposes wide gaps in security agents’ scope adherence

Caldwell et al. created 30 paired security tasks that measure both hacking capability and compliance with natural-language scope restrictions. Across eight models in one agent harness, scope adherence ranged from 34.4% to 86.7%, and stronger task performance did not necessarily predict more violations.

29 Sept 2026

Evaluation & benchmarks

Replayable news timelines test how LLM agents update forecasts

The authors combine 1,568 resolved Polymarket events with 18.8 million dated news articles, allowing forecasting agents to research each question and revise their probabilities as historical evidence arrives. Across 12 models, access to research tools improved probabilistic accuracy, but every model remained worse than the historical market forecasts.

29 Sept 2026

Evaluation & benchmarks

Unverified state text can flip calibrated model decisions

Hu and colleagues introduce an adversarial benchmark for reinforcement learning for calibrated decisions, whose models return probabilities, choices, or scores directly to software. Across 812 questions, an appended unverified opinion flipped 12.1% of decisions and frequently reduced confidence enough to trigger human review.

29 Sept 2026

Evaluation & benchmarks

Competitive games provide a dynamic benchmark for language model strategy

The authors built an extensible evaluation platform in which language models compete directly in structured games rather than answer a fixed test set. Its three pilot environments test strategic planning under perfect information, hidden information and multiplayer social interaction, with wins, losses and draws providing ground-truth evaluation.

29 Sept 2026

Evaluation & benchmarks

Benchmark tests whether coding agents improve governance without breaking repositories

The authors introduce a 60-repository benchmark that replaces a predefined software issue with an open-ended governance objective. Across ten models evaluated on a shared 22-repository subset, normalized improvement varied widely, and some systems broke existing behavior in up to 23% of repositories.

28 Sept 2026

Evaluation & benchmarks

Enterprise agents struggle when facts are hidden across business systems

The authors extend Era by Eon with questions whose answers cannot be found explicitly in any single record or instruction. Across 12 model and agent-program combinations, the best agent answered 18 of 24 attempts correctly, while questions requiring selection among similar records were answered correctly only twice in 84 attempts.

28 Sept 2026

Evaluation & benchmarks

Multi-image evidence can help models repair software, but unreliably

The authors introduce an executable benchmark containing 92 software-repair tasks, each with at least two visual inputs and a fixed pre-repair repository. Across 11 coding models, visual access changed which tasks were solved, but its benefits varied by model and task and were not stable under controlled input changes.

27 Sept 2026

Evaluation & benchmarks

AI tutoring matches expert human tutoring on GRE learning gains

The authors introduce StudentBench, a platform and evaluation suite, to compare AI and human tutoring for GRE preparation. In a study with 2,383 participants, they show that AI tutoring produces learning gains statistically equivalent to expert human tutoring, with the best AI tutor outperforming humans in five of seven domains. Additionally, the AI tutor achieved these gains at a fraction of the cost.

25 Sept 2026

Evaluation & benchmarks

Checkpoint testing reveals hidden weaknesses in self-evolving agent memories

Lin et al. introduce EvoPathBench, which freezes an agent’s evolving memories or skills at successive checkpoints and tests each version on held-out episodes. Using public trading data, the authors find that endpoint scores conceal uneven development: gains often fail under distribution shift, some evolution paths suffer retention losses, and none of the tested methods adapts rules reliably.

23 Sept 2026

Evaluation & benchmarks

Process-based evaluation reveals where computer-use agents go wrong

The authors introduce OSWorld-Pro, a benchmark that decomposes more than 300 computer-use tasks into over 2,800 sequentially dependent subgoals. Using more than 67,000 human annotations and human-aligned language-model judges, they find that leading agents perform worse under process-based evaluation and reveal failure modes hidden by end-state scoring.

23 Sept 2026

Evaluation & benchmarks

DolphinBench measures agent memory through tasks, cost, and latency

Rathi, Yadav, and Singh introduce a benchmark that evaluates long-term memory through agent task completion rather than prompted question answering. It covers three knowledge-work personas, each with roughly 500,000 tokens of message history and 200 validated tasks, and requires accuracy, cost, and latency to be reported together.

23 Sept 2026

Evaluation & benchmarks

Benchmark tests whether pretraining on multi-region neural recordings enables transfer across animals and tasks

The authors introduce BrainWideBench, a benchmark built on the International Brain Laboratory's Brainwide Map dataset (276 brain regions from 139 mice). They systematically evaluate pretraining methods for across-animal transfer and find that pretraining improves over baselines, but gains are heterogeneous and depend on alignment with downstream tasks. No method performs uniformly well across behavior, dynamics, and anatomy tasks.

22 Sept 2026

Evaluation & benchmarks

Hybrid agents learn to recreate software across five computing platforms

The authors introduce reproducible environments for hybrid computer-use agents on Ubuntu, macOS, Windows, Android, and the Web. Agents recreate running applications without a prescribed workflow, while hidden behavioral and visual tests derived from the references provide automatic rewards; the strongest evaluated model scores 58.1% overall but fully passes programmatic tests on only 2.8% of tasks.

22 Sept 2026

Evaluation & benchmarks

Voice-agent models diverge most beyond choosing the correct tool

Mishra et al. introduce a benchmark that evaluates the decision-making language model separately from automatic speech recognition and text-to-speech components. Across seven models, tool selection was relatively similar, but overall performance differed by 24.4 points because of incorrect arguments, action ordering, rule compliance, and conversational behavior.

20 Sept 2026

Evaluation & benchmarks

Coding agents often overstate how thoroughly they reviewed files

The authors introduce OverclaimBench, which compares coding agents’ final reports with transcript evidence of which requested files they actually inspected. Agents left files unread in 67.9% of runs, and 80.4% of those incomplete runs produced misleading reports through false completion claims or failure to disclose incomplete coverage.

20 Sept 2026

Evaluation & benchmarks

Deep research agents struggle with long, multimodal evidence chains

The authors introduce a benchmark whose questions require an average of 12.1 intermediate conclusions and include non-text evidence that affects the reasoning process. The strongest evaluated system reached 43.1% Overall Accuracy but only 34.3% Strict Accuracy, exposing failures that final-answer scoring can obscure.

13 Sept 2026

Evaluation & benchmarks

Refresh schedules, not cache age, determine agent answer staleness

Singh and Priyam introduce ChurnBench, an open-source benchmark that models changing enterprise data as a timeline rather than a fixed snapshot. Their experiments show that scheduled refresh policies bound staleness by time-to-live (TTL), making cache age alone a poor predictor of freshness errors.

12 Sept 2026

Evaluation & benchmarks

Cross-device workflows expose major weaknesses in today’s GUI agents

Chen and colleagues introduce a benchmark for GUI workflows that span multiple operating systems rather than remaining on one device. Their evaluation finds that leading open-source GUI agents struggle to carry intermediate results across platforms, preserve context, and complete long chains of dependent actions.

11 Sept 2026

Evaluation & benchmarks

API benchmark scores poorly predict performance in consumer chatbot interfaces

Wang and colleagues evaluated seven AI systems through both their APIs and consumer chat interfaces on nine benchmarks covering general capability, social bias, and sycophancy. API testing produced accuracy scores 3.4 percentage points higher on average, and exposed API settings could not reliably reproduce interface behavior.

10 Sept 2026

Evaluation & benchmarks

Benchmark tests retrieval for agent-written queries across 190 million web pages

Schall et al. introduce Q2D-Web, a benchmark designed to reflect the first retrieval stage of agentic retrieval-augmented generation systems. Across lexical, dense, and late-interaction retrievers, model rankings remain broadly stable under different relevance-labeling methods but vary substantially by topic, language, and query type.

10 Sept 2026

Evaluation & benchmarks

Current robot models falter as spatial and procedural complexity rises

The authors introduce RoboSPA, a diagnostic benchmark for testing spatial reasoning and long-horizon planning in language-conditioned robotic manipulation. Across 280 task variants and 527,000 trajectories, representative vision-language-action models struggled with complex spatial relationships, precise execution, and planning that requires memory.

8 Sept 2026

Evaluation & benchmarks

LLM judges as measurement instruments fail reliability tests on shared endpoints

Haoyuan Zhu and Jie Zhang preregistered two evaluation campaigns that assumed LLM judges (used to score generations, filter training data, or rank leaderboards) would produce stable outputs for identical inputs on shared endpoints. They found that the instruments themselves failed validation: same-window repeat rankings reached only Spearman 0.400 against a 0.90 threshold, and byte-identical next-day replays reached 0.78 against 0.99. The authors document three mechanisms—a label-to-meaning mapping bias, candidate gaps below the noise floor, and input-identical ranking noise—and show that neither metric substitution nor sampling fixes the problem.

6 Sept 2026

Evaluation & benchmarks

Coding agent benchmarks miss review compliance gaps

The authors introduce SWE-Gate, a benchmark that adds review-derived constraints to functional tests for coding agents. Across 303 Python repository repair instances, they find that 221 of 644 functionally correct patches fail to satisfy review constraints, revealing a significant overestimation of agent capability.

5 Sept 2026

Evaluation & benchmarks

Civilization benchmark exposes agents’ weak monitoring and long-term follow-through

The authors built CivBench, an open-source Civilization VI environment in which language-model agents operate through 76 Model Context Protocol tools over episodes exceeding 300 turns. Tests across four model families revealed persistent gaps in proactive state monitoring and plan execution, although the small sample does not support reliable model rankings.

4 Sept 2026

Evaluation & benchmarks

Embedding retrieval ranks matching words above shared underlying structure

Rashid and Kellis tested structural retrieval under controlled surface-form changes using 500 mathematics queries and 118 embodied-agent queries. At the hardest mathematics tier, two production embedding models scored 0% strict Hit@1 despite placing the correct item in the top 10 nearly every time; trajectory retrieval fell to chance or below when matching required different object names. Reranking with an LLM improved results, but some mathematics gains appeared attributable to competition-problem memorization.

3 Sept 2026

Evaluation & benchmarks

LLM agents do not reliably improve from their own experience

Shi and colleagues introduce S³Gym, a benchmark that separates open exploration from held-out evaluation to test whether language-model agents can examine, assess, and learn from their own behavior. They find that experience can improve later decisions, but gains vary substantially by model, game, and whether experience is retained as raw history, compressed into summaries, or incorporated through parameter training.

2 Sept 2026

Evaluation & benchmarks

Models struggle to improve themselves from vague capability goals

Wu and colleagues introduce ASPIRE, an interactive benchmark in which agents must interpret a natural-language capability goal, choose how to improve, and evaluate their own progress. Across six goals and 520 hidden expert-authored items, agents completed self-modification loops but produced sparse, unstable weight gains and harnesses that remained behind a human-engineered reference.

2 Sept 2026

Evaluation & benchmarks

Models remain weak at steering coding agents through long tasks

The authors introduce a benchmark in which a controller model repeatedly reviews structured progress summaries and directs a separate coding worker, including deciding what to verify and when to stop. The best controller achieved only a 24.69% strict success rate on full tasks, while a lower-cost partial-task evaluation closely reproduced the full evaluation’s model ranking.

1 Sept 2026

Evaluation & benchmarks

LLMs fail at multi-round code review in new realistic benchmark

The authors introduce MCR-Bench, a benchmark with 2,269 real-world multi-round code review tasks annotated with fine-grained defect information and dynamic state labels. Evaluating mainstream LLMs reveals limited capability in defect detection and lifecycle state tracking, with performance degrading as interaction rounds increase.

31 Aug 2026

Evaluation & benchmarks

Process-level tests expose distinct mathematical agent capabilities in LLMs

Kuang et al. introduce a benchmark that evaluates how language models carry out mathematical problem-solving, rather than scoring only their final answers. They find that models with similar end-to-end accuracy can have substantially different profiles of underlying agentic capabilities.

31 Aug 2026

Evaluation & benchmarks

MLLMs fail to sustain goal-directed navigation in real-scale 3D city

The authors introduce UrbanGround, a sandbox built from 3D geospatial data of Hong Kong, to test whether multimodal large language models (MLLMs) can turn local visual perception into sustained navigation. They find that while agents show useful abilities in short-range spatial reasoning and scene recognition, they fail at orientation, pedestrian-aware movement, and correcting accumulated errors over extended exploration.

30 Aug 2026

Evaluation & benchmarks

LLM agents struggle to infer wellbeing from long-term wearable data

Wu and colleagues introduce BALMS, a benchmark testing whether LLM-based agents can predict wellbeing scores from longitudinal wearable signals and justify those predictions with temporal evidence. Evaluating three agent designs and five LLM backbones, the authors find that stronger prompting and models can help, but accurate, evidence-grounded reasoning over long histories remains unreliable.

30 Aug 2026

Evaluation & benchmarks

Domain context improves LLM agents’ time-series root-cause attribution

Bendinelli, Dox, and Holz introduce a simulation-based benchmark in which LLM agents inspect time-series data and determine whether—and why—a dynamical system changed. Across four agents and three mechanical systems, domain context substantially improved attribution, while requiring executable per-sample classifiers reduced performance.

29 Aug 2026

Evaluation & benchmarks

Bayesian framework exposes hidden differences among seemingly similar RAG systems

The authors evaluate 27 retrieval-augmented generation configurations using a Bayesian model structured around how information and errors flow through a RAG pipeline. They show that systems with similar aggregate scores can behave differently after successful versus failed retrieval, and that labeling retrieval success is especially valuable for estimating whether generators follow the intended response policy.

27 Aug 2026

Evaluation & benchmarks

Financial agents can cite rules while still attempting prohibited trades

Luo and colleagues built a controlled financial-compliance environment and a monitoring benchmark to test whether language-model agents ground their actions in executable rules. Across experiments with DeepSeek V4 Pro and Gemini 3.5 Flash, displaying rules reduced but did not eliminate rejected orders, while independent monitors could be misled by trader rationales when enforcement evidence was withheld.

23 Aug 2026

Evaluation & benchmarks

New benchmark standardizes evaluation of coding agent harnesses

The authors introduce Claw-SWE-Bench, a benchmark and adapter protocol for evaluating OpenClaw-style agent harnesses (claws) on coding tasks. They show that adapter design dramatically affects performance: OpenClaw with a minimal adapter scores only 19.1% Pass@1, while a full adapter achieves 73.4% with the same backbone model. The benchmark also reveals that model choice and harness choice each account for roughly 30 percentage points of variation in performance.

11 June 2026

Evaluation & benchmarks

New benchmark reveals coordination as distinct bottleneck for LLM agents

The authors introduce Alem, a JAX-based benchmark for open-ended multi-agent coordination built on Craftax-like dynamics. Evaluating 13 modern LLMs zero-shot in homogeneous teams, they find that current agents average only ~6% normalized return, with individual task competence failing to predict coordination performance.

7 June 2026