Evaluation & benchmarks

New benchmarks, evaluation methodology, measuring what models can do

97 articles · page 1 of 3

All Agents Language models Reinforcement learning Vision Generative media Speech & audio Robotics & embodied AI Safety & security Evaluation & benchmarks Efficiency & systems Code & software AI for science & health Theory

Viewgrid list compact

LLMs fail multi-step math despite solving each step correctly

Wang et al. introduce OracleLadder, a diagnostic tool that provides increasing levels of oracle help to large language models on multi-step math problems. By giving models a roadmap of sub-goals and their answers, they isolate failures into five categories. The dominant failure mode across all tested models is a composition gap: models can solve each intermediate step alone but cannot compose them into the correct final answer.

3 Oct 2026

Evaluation & benchmarks

LLM judges often encode better preferences than their verdicts reveal

Mukherjee and Sitaram test whether an incorrect LLM judgment reflects missing knowledge or a failure to express information already represented inside the model. Across 64 open-weight evaluators and 14 datasets, they find a substantial internal-output gap on pairwise tasks with misleading surface cues, but little benefit from internal read-outs when those cues are absent.

3 Oct 2026

Evaluation & benchmarks

New benchmark tests automated theorem proving for research-level theoretical computer science

The authors introduce TCSAlgBench, a benchmark and reusable pipeline for natural-language proof discovery in theoretical computer science. It consists of 398 theorem-level challenges from 138 papers from STOC and COLT 2026. In their evaluations, the best-performing system (GPT-5.6 Sol max with discussion) achieved 23.6% verifier-accepted coverage after 10 rounds, while an agentic planning workflow reached 25.4%.

3 Oct 2026

Evaluation & benchmarks

Video-language models often bind actions to the wrong people

Faure and colleagues introduce ActionLens, a 6,701-question video benchmark designed to test whether vision-language models can associate an action with the correct person and moment. Across 20 models, the strongest system reached 68.8% accuracy overall, while on a human-reviewed subset it scored 65.9% against a 91.0% pooled human reference.

3 Oct 2026

Evaluation & benchmarks

New benchmark tests if AI can track objects after they vanish from view

Ma et al. introduce Beyond3D, a VQA benchmark with 9,000 questions over 135 egocentric videos that tests whether VLMs can track an object's location after it is moved and has left the field of view. The best model achieved only 42.2% accuracy, with the largest failures in recovering when an object was last visible, indicating that current models struggle with out-of-sight spatiotemporal reasoning.

3 Oct 2026

Evaluation & benchmarks

Benchmark exposes distinct weaknesses in multimodal audio-video generation systems

The authors introduce ORAV Bench, a benchmark for generating audio and video from several heterogeneous references linked by a textual instruction. Testing five frontier systems shows that aggregate rankings obscure composition-specific strengths, while models frequently copy unintended source content instead of transferring only the requested attributes.

3 Oct 2026

Evaluation & benchmarks

Surrogate benchmarks enable systematic evaluation of scaling analysis methods across model types

The authors present two surrogate benchmarks, ScAn-Bench-LLM and ScAn-Bench-VLM, built from 4524 and 8024 checkpoints of language and vision-language model pipelines, respectively. They use these to perform the first systematic comparison of scaling analysis methodology across data modalities, covering both data acquisition and extrapolation approaches.

3 Oct 2026

Evaluation & benchmarks

Indic-centric corpus of 1.16 million sentence pairs improves machine translation for Indian languages

The authors introduce COILD, a parallel corpus of 1.16 million human-verified sentence pairs covering 20 Indian language pairs from four language families. Fine-tuning IndicTrans2 and NLLB-200 on COILD produced higher translation quality across all metrics and domains.

3 Oct 2026

Evaluation & benchmarks

AI agents reproduce only 41% of machine learning papers with full code and weights provided

The authors introduce RECLAIM, a benchmark of 100 NeurIPS 2025 papers with predefined reproduction targets and GPU-hour budgets. They test four AI agents once per paper; the best agent reproduces 41% of Run-tier papers (code, data, weights provided), 27% of Retrain-tier (no weights), and 15% of Reimplement-tier (no code). Failed attempts use only 29% of their budget on average, and the most common error is implementing methods without verifying against the paper's reported numbers.

3 Oct 2026

Evaluation & benchmarks

Video models struggle to connect cinematic techniques with storytelling intent

Xing and colleagues introduce CinematicVQA, a benchmark for reasoning about how cinematographic choices shape what viewers perceive and how a scene functions narratively. Their evaluations reveal a semantic gap: large vision-language models describe visible presentation more reliably than they identify the filming techniques behind it, while targeted fine-tuning improves narrative and multi-step reasoning.

3 Oct 2026

Evaluation & benchmarks

LLMs struggle to match multidisciplinary tumor board discussions in new benchmark

The authors introduce OpenTumorBoard, a benchmark derived from 12,534 minutes of publicly available tumor board recordings on YouTube, comprising 611 patient cases and 19,157 discussion turns across ten specialist roles. Evaluating 14 general-purpose and medical LLMs, they find that the best models achieve modest scores of 3.43/5 on responding to individual specialist questions and 2.78/5 on generating entire board discussions that align with recorded consensus. Supervised finetuning and reinforcement learning on the benchmark data improve performance on a held-out test set.

3 Oct 2026

Evaluation & benchmarks

Standard benchmarks overstate the accuracy of reused language model caches

Cestola et al. examine position-independent key-value cache reuse, a technique intended to accelerate retrieval-augmented generation by reusing cached representations of text chunks at different prompt positions. They find that common aggregate accuracy metrics and existing datasets can conceal the accuracy loss caused specifically by reuse, then introduce Boxoffice to generate more demanding evaluations.

3 Oct 2026

Evaluation & benchmarks

New benchmark measures whether AI systems can explore, not just recall, unfamiliar rules

The authors introduce ExplorationBench, a benchmark that evaluates AI systems' ability to explore unknown environments by forming hypotheses, designing experiments, and iterating on results. Its two executable sandboxes make every answer exactly checkable while ensuring that pretraining recall alone cannot solve the tasks. Evaluating ten AI systems, the authors find that the strongest can acquire and apply these unfamiliar rules, though performance varies across trajectories and continued exploration can stall or reverse earlier gains.

3 Oct 2026

Evaluation & benchmarks

Coding agents nearly finish repositories but still miss explicit requirements

Yang et al. introduce Zero2Repo, a benchmark that gives coding agents an empty workspace, a product requirements document and an interface contract, then asks them to deliver an entire software repository. On 11 tasks, the strongest evaluated agent completed 10, while failures were usually nearly correct implementations undone by one omission or uncommon rule.

2 Oct 2026

Evaluation & benchmarks

Voice agents lose document grounding as conversations and context grow

Mathur et al. introduce DSB-DG, a benchmark for testing whether real-time voice agents answer faithfully from supplied documents. Across cascaded and speech-to-speech systems, the authors find that grounding deteriorates under longer contexts and sustained dialogue, with failures often producing unsupported claims rather than explicit abstention.

2 Oct 2026

Evaluation & benchmarks

Top AI agents complete fewer than one-quarter of professional tasks

Finley and colleagues introduce DAYJOB, a benchmark of 130 long-horizon assignments designed by professionals in healthcare and finance. Across 30 model configurations, even the strongest system passed fewer than one-quarter of attempts, while typical configurations rarely produced deliverables satisfying every required criterion.

2 Oct 2026

Evaluation & benchmarks

Agent performance depends more on system configuration than model size

Wiedmann and colleagues evaluate coding agents as configurable systems rather than treating the underlying language model as the sole unit of analysis. They find that task information has a larger effect than either model size or time budget, while repeated runs of identical configurations still produce substantial variation.

2 Oct 2026

Evaluation & benchmarks

Benchmark exposes gap between choosing tools and using them

The authors introduce an 18-task benchmark that evaluates humanoid tool use as an end-to-end process, rather than treating recognition, manipulation and locomotion separately. Across seven policies in simulation and three on a Unitree G1 robot, they find that selecting an appropriate tool does not reliably translate into successful task completion.

2 Oct 2026

Evaluation & benchmarks

Language agents rarely withdraw selectively when instructions or permissions change

Abouzahra introduces NAQD-Env, a benchmark for testing whether language agents revise plans selectively when evidence, authorization, or constraints change. Across 3,150 model-prompt episodes, withdrawal recall never exceeded 0.06, no valid resumption occurred when eligible, and only one episode fully matched the benchmark’s reference policy.

1 Oct 2026

Evaluation & benchmarks

Clinical language models diagnose well but often mishandle full encounters

Fang et al. introduce a clinician-authored benchmark that evaluates the process of clinical assessment rather than diagnosis from a complete case summary. Across 31 models, diagnostic accuracy reached 90.7%, yet even the strongest systems passed fewer than 30% of tasks when required to question a virtual patient, request appropriate examinations, respect constraints and provide the correct diagnosis.

1 Oct 2026

Evaluation & benchmarks

Self-updating agents can improve performance while quietly weakening safety

Das et al. introduce SEABench, a benchmark for testing whether agents that modify their own operating harness develop unsafe behavior over time. Across 48 longitudinal task sequences, self-evolution improved task completion but also produced safety regressions attributable to earlier updates, despite the absence of direct adversarial intervention.

1 Oct 2026

Evaluation & benchmarks

Shopping agents falter when user needs change across a conversation

Yang et al. introduce RealWorldShop, a benchmark designed to evaluate the full decision process of conversational shopping rather than isolated replies or final purchases. Their experiments indicate that current agents often sound plausible turn by turn but fail to preserve user constraints and converge on catalog-grounded recommendations across a session.

1 Oct 2026

Evaluation & benchmarks

Security agents lose evidence support when available telemetry changes

The authors introduce APTInvestBench, a benchmark that tests whether LLM agents can investigate advanced persistent threats when the available security logs change. Across eleven models, agents retrieved sufficient evidence for 44.3% of recoverable attack actions on average, but their final citations adequately supported only 25.0%, revealing substantial losses between finding evidence and reporting it.

1 Oct 2026

Evaluation & benchmarks

Frontier vision-language models mostly fail a real low-speed driving course

Ramabadran, Mahns and Gessler tested whether general-purpose vision-language models could drive a real car without driving-specific training or a conventional autonomy planner making high-level decisions for them. Across four frontier models and up to three attempts each, only GPT-6 Astra finished the course, while every other attempt covered less than half of it.

1 Oct 2026

Evaluation & benchmarks

Long context windows do not ensure reliable sustained execution

Willette, Puvvada and Ginsburg introduce Long-Transduction, a diagnostic for whether models can repeatedly read state, transform it and emit correctly aligned outputs over long generations. Their experiments show that advertised context capacity does not imply dependable execution: accuracy fell substantially as context grew from 4K to 128K tokens, even though the underlying operations remained controlled.

1 Oct 2026

Evaluation & benchmarks

Correcting search bias makes deep research conclusions more representative

Xiao and colleagues treat an agent’s search process as adaptive evidence sampling, where early results influence later queries, document choices and stopping. Their Causal Evidence Selection Correction method, CESS, better estimated the overall direction of an available document pool than simply averaging the documents an agent read.

1 Oct 2026

Evaluation & benchmarks

Clinical agents falter when health records cannot support the question

Qiao et al. built an interactive benchmark from MIMIC-IV records to test clinical agents on paired questions that are either supported by the database or rendered unanswerable by a single evidence discrepancy. Across proprietary and large open-weight models, average task success fell from 62.2% on supported questions to 37.9% on noisy counterparts, showing that retrieval and query execution do not ensure evidence-grounded reasoning.

1 Oct 2026

Evaluation & benchmarks

New benchmark disentangles sycophancy from empathy in multi-turn LLM conversations

The authors introduce FIGS, a benchmark with 500 multi-turn scenarios and an automated judge that separately scores sycophancy (factual yielding) and calibrated validation (appropriate empathy). Evaluating leading LLMs, they find a consistent trade-off: models gradually become sycophantic over sustained conversation, or shift to robotic dismissal, showing that balancing truthfulness with supportiveness remains unsolved.

1 Oct 2026

Evaluation & benchmarks

Sparse Human Label Overlap Undermines Reliable LLM Judge Validation

Li, Mukherjee and Pal study how reliably teams can validate an LLM judge when only a small fraction of items receive labels from both humans and the model. Through statistical analysis, simulations and tests of 10 judges, they find that sparse overlap can produce high error rates even when the overall annotation budget appears substantial.

30 Sept 2026

Evaluation & benchmarks

Fixed LLM judges can mismeasure upgrades as agents change

Li tests whether a language-model judge calibrated on one agent version remains reliable when comparing newer versions. Across SWE-bench, tau-bench and expert-labeled trajectories, fixed judges showed version-dependent errors that could distort estimated improvements, while auditing current paired outputs proved safer than reusing old calibration.

30 Sept 2026

Evaluation & benchmarks

Models struggle to predict software behavior after stateful interventions

Xinran Zhang introduces CTE-Bench, an executable benchmark that isolates whether models can predict the downstream effects of modifying a stateful software service. Across four API-hosted models, exact response accuracy on intervention-affected calls reached 54.3% to 61.5% when correct previous responses were supplied, but fell below 34% without that feedback or when models relied on their own predictions.

30 Sept 2026

Evaluation & benchmarks

Coding agents struggle to turn software proposals into working changes

Peng et al. introduce LoLBench, a benchmark that asks coding agents to implement substantial software enhancements from human-written proposals rather than detailed coding specifications. The results separate two challenges: understanding how a high-level design maps onto an existing repository, and correctly implementing the resulting changes. Even the strongest evaluated agent resolved only 14 percent of tasks.

30 Sept 2026

Evaluation & benchmarks

Conversational graph querying exposes a gap in model autonomy

Zhang et al. introduce CypherTurn, a benchmark for generating Cypher graph queries across conversations in which later requests depend on earlier results. Testing 15 models under guided and autonomous protocols, they find that turn-level competence does not translate into reliable sessions, and that managing errors is partly distinct from generating correct queries.

30 Sept 2026

Evaluation & benchmarks

Correct Answers Often Fail to Prove Genuine Multi-Hop Reasoning

Tao et al. audit multi-hop question-answering benchmarks by removing evidence for individual reasoning steps and checking whether models can still recover answers they previously got right. Existing benchmarks show weak dependence on their annotated chains, while the authors' REALHOP framework raises that dependence substantially and better separates model performance.

30 Sept 2026

Evaluation & benchmarks

LLM agents deploy repairs better than systems built from scratch

Ding et al. introduce 136 deployment tasks evaluated through programmatic checks in freshly rebuilt environments, without an LLM judge. Seven models resolved 52.9 to 75.0 percent of tasks, with substantially stronger results on repairing existing configurations than on creating new ones.

30 Sept 2026

Evaluation & benchmarks

Generating user guides can improve web agents’ action selection

Gan et al. built WebMRE from successful WebArena trajectories to study web-agent training without the variability of live websites or model-based judges. Their experiments suggest that generating a human-readable guide can help a model choose the corresponding interface action, although the gains from joint training were small and depended on model scale and output order.

30 Sept 2026

Evaluation & benchmarks

Agent defenses leave privileges unseen by standard security benchmarks

Sharma et al. built Ajar to measure open privilege, meaning tool actions an agent defense would allow even though a task does not require them. Applied to five defenses on AgentDojo, it revealed differences that could not be inferred from attack success rates or benign task completion.

30 Sept 2026

Evaluation & benchmarks

Marketplace Steering Often Diverts Computer Agents From Users’ Best Choices

Li and colleagues introduce CAVEAT, a benchmark testing whether computer-use agents preserve user objectives when online marketplaces favor competing outcomes. Across five model families, steering reduced the user-optimal purchase rate from 78.6% to 17.3%, while a targeted agent harness recovered as much as 80.0 percentage points.

30 Sept 2026

Evaluation & benchmarks

Fixed controls expose limits in financial LLM agent evaluation

Xue tests whether common performance and auditing scores support the conclusions drawn from them in a financial agent harness. Fixed-response replay produced a narrower estimate of execution stress than independent runs, while zero-, one-, and two-defect controls revealed that apparently strong audit recall could coexist with low precision and poor exact identification.

30 Sept 2026

Evaluation & benchmarks

Email agents execute valid tool calls but often miss the task

Singh et al. introduce EmailBench, a self-contained benchmark for testing agents on email and productivity workflows involving retrieval, temporal reasoning, and structured actions. Evaluating eight language-model configurations, they find a large gap between successful API execution and successful completion of the requested task.

30 Sept 2026

Evaluation & benchmarks

Sparse annotation overlap makes LLM judge validation decisions unreliable

Li, Mukherjee and Pal analyze how limited overlap, the proportion of items labeled by multiple raters, affects validation of LLM judges against humans. Across theoretical analysis and experiments with 10 judges, they find that sparse overlap sharply increases erroneous approval, rejection and ranking decisions, while stratified allocation can improve reliability without adding annotations.

30 Sept 2026

Evaluation & benchmarks

Adaptive testing exposes major security repair gaps in coding agents

The authors developed SecProbe, an evaluation framework that adaptively selects or synthesizes repository-scale vulnerability-repair tasks using Item Response Theory, a statistical method for matching test questions to ability. Across nine frontier models and two agent harnesses, performance remained low, while adaptive testing produced comparable ability estimates with fewer attempted tasks than baseline evaluation strategies.

30 Sept 2026

Evaluation & benchmarks

Search agents struggle to find broad evidence for open research problems

Ding et al. introduce a benchmark in which agents must search an open literature corpus without knowing the later target paper or its cited antecedents. The strongest evaluated agent reaches 0.284 nDCG@10, with the largest observed loss occurring before selection, because relevant prior work is never surfaced.

30 Sept 2026

Evaluation & benchmarks

Task-aware certificates safely automate part of LLM agent evaluation

Gan et al. study how much human review an LLM judge can replace while keeping errors among automated decisions below a specified budget. Their task-level bootstrap certificate avoided the false assurances produced by standard independent-sample methods, and certified 30% to 59% automation for a trained 4B judge at a 10% error budget.

30 Sept 2026

Evaluation & benchmarks

Benchmark tests agent control of anomaly detectors under delayed feedback

Han et al. introduce an executable benchmark that makes an agent choose among 11 detectors across eight chronological deployment windows, with outcomes revealed one window later. Across seven temporal graph datasets, the protocol produces auditable trajectories showing when controllers respond usefully, ineffectively, or at excessive computational cost to delayed evidence.

30 Sept 2026

Evaluation & benchmarks

LLMs often diagnose without correctly valuing available medical evidence

Feng et al. tested nine language models on MedEVM, where medical evidence arrives sequentially and models must choose whether to wait or diagnose. The authors find that models often submit diagnoses at the wrong evidential moment, remain sensitive to evidence order and misleading details, and gain 12.0 to 51.1 percentage points in accuracy when an evidence-verification harness controls submission.

30 Sept 2026

Evaluation & benchmarks

GUI agents sacrifice future options in long visual puzzles

Zhang et al. evaluate general-purpose GUI agents on 114 levels across six visual puzzle games, using persistent boards where early moves can create unannounced dead ends. Performance declines sharply as decision chains grow longer, and diagnostic experiments suggest that rules, state hints, and memories of prior failures do not resolve the central planning weakness.

30 Sept 2026

Evaluation & benchmarks

Faulty harnesses can radically distort prompt injection security results

Shaw audits a benchmark for indirect prompt injection, where malicious instructions hidden in retrieved data redirect an agent's tool calls. Re-scoring the same execution traces at the tool-argument level reduced measured attack success from 21.7% to 1.2%, showing that evaluation defects can dominate reported security results.

29 Sept 2026