Evaluation & benchmarks

New benchmarks, evaluation methodology, measuring what models can do

456 articles · page 3 of 10

Evaluation & benchmarks

Backtesting method lets AI agents forecast past events for instant feedback

The authors introduce FABench, a backtesting environment for forecasting agents that replaces live benchmarks with resolved events. By generating questions from historical data and restricting agents to information available before the forecast date, the system provides immediate, reproducible feedback. Evaluating eight models, the authors find that retrieval consistently improves accuracy, with top agents matching market consensus on 54 prediction-market questions.

Evaluation & benchmarks

Small language models judge outputs more accurately when trained on counterfactual pairs

The authors introduce Counterfactual Pairwise Augmentation (CPA), a training approach that boosts the accuracy of small language models used as pointwise judges in agentic systems. By generating and auditing a counterfactual response with the opposite label for each pointwise training item, CPA creates pairwise training data that isolates the decision boundary. Fine-tuning a Qwen3-8B model jointly on pointwise and these pairwise examples improved criterion-averaged macro-F1 by 2.5 points over pointwise-only training, and outperformed the model merging strategy of Prometheus 2.

Evaluation & benchmarks

Language models agree more with reasoning, but can share more mistakes

Mila Gorecki and Moritz Hardt find that enabling reasoning makes language models more likely to agree when classifying the same individuals, even after accounting for accuracy differences. But individuals who elicit longer reasoning traces receive less consistent predictions across models. The distinction matters: changing a reasoning setting and observing a long response answer different questions.

Evaluation & benchmarks

Training data attribution for AI language models degrades with redundancy and distractors

The authors introduce CATT, a synthetic benchmark for evaluating data attribution methods in language models, where ground truth is unambiguously known. They find that document redundancy and lexical distractors consistently degrade attribution performance, and that standard techniques wash out localized causal evidence by averaging across all tokens. Applying negative log-odds gradients and aggregating only the top token scores improves recall by 35 to 72 percentage points across multiple methods.

Evaluation & benchmarks

AI agents that succeed at tasks may still fail their mid-way steps

Rezaeian et al. propose a trajectory-level evaluation protocol for tool-using AI agents that measures whether every prefix of a tool-call sequence remains executable under tool schemas, state preconditions, and data dependencies. Applying it across multiple benchmarks, they find that outcome-only rankings can diverge sharply from prefix-level reliability: unverified plan-conditioned agents often achieve higher final success while substantially lowering executability metrics, whereas pre-execution plan checking improves executable-prefix survival without sacrificing terminal performance.

Evaluation & benchmarks

Batching makes AI grading verdicts depend on the other items

Peng Xie and Amr Alanwar tested whether language models grade items independently when many items are placed in one prompt. They found that changing which items share a batch flips 11-22% of pass/fail verdicts on harm detection and short-answer grading, far more than changing instruction wording or item keys. The instability is concentrated on ambiguous items and persists even when averaging over multiple batch orders.

Evaluation & benchmarks

AI agents that satisfy more search constraints recommend fewer places

The authors introduce ATLAS-Bench, a benchmark of 1,200 local search queries that distinguish mandatory from optional and objective from subjective requirements, along with a reproducible sandbox called ATLASBox. Testing 14 configurations of 11 LLMs, they find a trade-off between constraint satisfaction and query coverage: the highest-accuracy agent (GPT-6-Luna) satisfied 71.5% of constraints in its answers but found an acceptable option for only 67% of queries, while a model with lower per-answer accuracy (GLM-5.3) succeeded on 85% of queries.

Evaluation & benchmarks

AI health overviews cite fewer top sources and often lack evidence

The authors evaluated Google's AI Overviews (AIOs) for 3,611 health questions, comparing their sources to traditional search results and assessing statement grounding and accuracy against physician-written rubrics. They found that AIOs cite only 44.4% of the top five traditional results, overrepresent commercial publishers like WebMD and Healthline at the expense of academic and government sources. An estimated 56.7% of cited statements were supported by their source, while 20.8% of all statements were factual claims with no citation; overall, AIOs scored 38.2% of available points on physician rubrics, and 22.2% of AIOs where a harmful claim was anticipated contained that content.

Evaluation & benchmarks

AI video narrators fit silences well, but pacing still trails human professionals

The authors built a professionally annotated dataset of 36 hours of user-generated videos and a reference-free evaluation framework for inline audio descriptions, rendering generated scripts to audio and measuring how well they fit natural silences. Testing nine frontier multimodal models, they find the best models describe visual events and avoid overlapping speech well, but human-written descriptions still have better narrative pacing. Ratings from blind and low-vision participants confirm the automated metrics: users narrowly prefer professional AD, yet find AI-generated AD highly beneficial for comprehension.

Evaluation & benchmarks

Visual scaffolds improve AI reasoning by bridging the reference gap, not perception

Multimodal large language models can count objects in an image but often double-count or miss items when reasoning across steps. Chen et al. show this failure is a 'reference gap', models cannot stably bind language to specific visual entities, and that adding task-appropriate external references like numbering or shading closes it, boosting accuracy by up to 25% without retraining.

Evaluation & benchmarks

Extra attempts help small language models solve terminal tasks, but not verify solutions

The authors decomposed the performance gap between small and frontier language models on terminal computer tasks into three capabilities: solvability (generating a correct solution with enough attempts), verification (selecting the correct solution from candidates), and confidence (calibrating success estimates). They found that repeated sampling narrowed the solvability gap but not the verification or confidence gaps, indicating that small models need better judgment, not just better generation.

Evaluation & benchmarks

Human egocentric video lifts robot grasp success from 16% to 58%

The authors introduce EgoContact, a dataset of 22.5K human interaction events with pixel-precise contact annotations. They find that current affordance and vision-language models struggle to predict these contact points, often failing even to localize the object. But replacing the memory bank of an existing affordance-transfer policy with EgoContact improves grasp success from 14.2% to 31.2% in simulation and from 16% to 58% on a real robot.

Evaluation & benchmarks

AI image models fail to combine multiple views of the same scene

Suhas et al. at the Max Planck Institute for Informatics introduce OneScene-Bench, a human-annotated benchmark that forces AI vision models to integrate evidence across multiple views rather than process each image independently. Testing twelve open-weight and proprietary models, the authors find that unique-object counting collapses when views are complementary, and that contradiction detection hovers at chance, while spatial reasoning is far stronger.

Evaluation & benchmarks

Proper coordinate alignment reshuffles novel-view AI rankings

The authors demonstrate that the evaluation of single-view novel view synthesis methods is flawed due to misaligned coordinate frames. They propose using 2D-to-2D and 2D-to-3D matching to properly align reconstructions, revealing that explicit 3D Gaussian methods outperform video-based ones, contrary to prior reports. A simple baseline using off-the-shelf video generation and geometry estimation achieves comparable performance without task-specific training.

Evaluation & benchmarks

Magic-trick videos expose the reasoning gap between AI models and humans

Shuo Wen and colleagues built The Prestige, a benchmark of 2,177 open-ended questions grounded in professional magic performances, and tested 14 leading vision-language models against human baselines. The best model scored 47.08%, versus 92.79% for humans, a 45.71 percentage-point gap. Models repeatedly failed to recall objects absent from the current frame, missed surprises that humans noticed, and struggled to reject false premises.

Evaluation & benchmarks

AI agent rankings shift under iterative feedback evaluation

The authors propose Seq@K, a metric that gauges a language model's ability to solve a task within K attempts when given feedback after each failure. Evaluating 16 frontier models on 7 benchmarks, they find that Seq@K rankings often diverge from standard Pass@K rankings, and cost-performance Pareto frontiers shift. This suggests that choosing a model based on feedback-blind accuracy may select the wrong one for iterative agentic workflows.

Evaluation & benchmarks

AI agents give plausible explanations but fail to recover scientific laws

The authors introduce SciGame, a benchmark that converts scientific laws into paired original and counterfactual environments. Testing five frontier models, they find that agents' familiarity with a law hurts performance when the underlying relation is altered, and that adaptive experimentation benefits only the strongest models. Even when agents provide good explanations, they frequently fail to recover the exact symbolic form of the law.

Evaluation & benchmarks

AI services secretly using cheaper models can be detected with few queries

The authors propose API-TAIL, a text-only audit that detects when an LLM API secretly routes some requests to a cheaper substitute. It scores each response under the claimed model, aligns one effective temperature to absorb batch-level shifts from legitimate decoding choices, then flags endpoints with too many remaining tail responses. On real OpenRouter endpoints it kept false alarms low while detecting 10% substitution in 13 of 16 substitute settings with 200 requests.

Evaluation & benchmarks

AI agents’ clicks marked wrong can work a third of the time

The authors propose evaluating GUI grounder clicks by whether they produce the same outcome as clicking the annotated target, rather than whether they land inside a bounding box. Using an automated probe that executes and fingerprints clicks on live web applications, they find that on a code forge and an online shop, 17.7% and 31.2% of clicks rejected by box scoring are actually outcome-correct. A blind audit of 11 models on ScreenSpot-v2 confirms the same pattern, with corrections of 0.2 to 1.8 points, as large as the gaps typically reported between models.

Evaluation & benchmarks

AI agents can reconstruct five years of tax records in minutes

Salapa introduces kw-bench, a benchmark built from real VAT reconstruction engagements, testing four agent harnesses on DeepSeek v4.1 Flash and Claude Opus 5.5. On DeepSeek v4.1 Flash, only the independent harnesses OpenCode and micro-cc succeeded, scoring 0.542 and 0.528; the lab harnesses Claude Code and Codex scored 0.00 each. On Claude Opus 5.5, micro-cc and Claude Code achieved 0.9986 and 1.000 in 22 and 25 minutes, collapsing a weeks-long engagement to minutes.

Evaluation & benchmarks

AI models struggle to report video tampering their internal features reveal

The authors introduce AV-FABLE, a dual-timeline benchmark for detecting sparse audio-visual manipulations in long-form videos. Zero-shot prompting of audio-visual large language models (AV-LLMs) achieves near-chance performance, but linear probes on their frozen representations recover strong forensic signals (90.8% audio AUROC, 80.8% visual AUROC). Building on this, the authors propose NFSR, which uses separate temporal decoders on frozen features to achieve state-of-the-art localization (91.30 AV-mAP, 96.04 detection AUROC in-domain).

Evaluation & benchmarks

AI moderators need different signals for human disagreement and costly mistakes

The authors audit a commercial System One content moderation model on the Civil Comments corpus, comparing it to 14 frontier reasoning models. The cheap model's zero-shot AUROC (0.83) is not statistically separable from 13 of the 14 frontier systems while costing 1–3 orders of magnitude less. However, the model's confidence conflates two distinct uncertainties, human disagreement and decision risk, which call for opposite escalation signals.

Evaluation & benchmarks

Default kernel metrics for generative models miss rotation changes at common sample sizes

The authors tested the default kernels of KID and KAD as permutation two-sample tests on real embeddings of symbolic music, audio and images, applying controlled changes (mean shift, isotropic contraction, rotation) matched in 2-Wasserstein distance. At 780 samples per side, both defaults detect matched mean shifts in every run but rarely detect the rotations, while sensitivity to contractions and rotations depends strongly on kernel bandwidth and sample size.

Evaluation & benchmarks

AI agents finish fewer than a quarter of five-stage professional tasks

The Continual Learning Exam (CLE) is a new benchmark for evaluating how LLM agents improve over time through accumulated experience. Created by over 50 domain experts, it comprises 120 tasks each with five increasingly difficult levels, spanning 89 professional roles. The top-performing system, Claude Code with Opus 5, completed only 47.7% of all levels, and completing all five levels in a task occurred in just 23.3% of tasks.

Evaluation & benchmarks

Coding agents can recognize risks and still take harmful actions

The authors introduce V3AST-Bench, a benchmark for evaluating the safety of LLM-based coding agents using executable checks over environment states, records, and artifact behavior. Across 86,496 trials with 11 models and five harnesses, even Claude Opus 4.8 with Claude Code caused verified harm on 20.9% of tasks. The results show that workflow capability alone does not guarantee safe execution, and that explicit risk recognition does not reliably prevent harmful actions.

Evaluation & benchmarks

Language models show hidden fairness failures when identity traits combine

The authors present ComFair, a configurable benchmark that systematically augments single-attribute fairness data with additional sensitive attributes to create 479,066 paired Dual- and Multi-attribute instances. Across ten LLMs and four task paradigms, they show that single-attribute evaluations often produce fragile conclusions: discriminative tasks reveal both aggravation (correct to wrong) and masking (wrong to correct) that aggregate scores conceal, generative tasks exhibit larger and directionally asymmetric changes, and model rankings vary with attribute composition.

Evaluation & benchmarks

Language models can understand others’ beliefs yet fail to act accordingly

Yin et al. introduce a multi-level evaluation of theory of mind that separates belief inference from its use in decision-making and task execution. Extending two verifiable tasks, household collaboration and calendar scheduling, with asymmetric information, they show that models often recover beliefs correctly yet produce suboptimal plans or fail long-horizon coordination. The findings challenge the assumption that story-based ToM benchmarks measure the full capability needed for collaborative agents.

Evaluation & benchmarks

AI agents rank similarly across board games, but software affects performance

Zhang and colleagues introduce DECE, a shared testing arena for language-model agents playing ten Eurogames, strategy board games built around persistent investments, contested resources and multiple scoring routes. Their experiments show that competitive rankings broadly carry across games, but performance also depends on the software running the model and does not consistently track token or monetary efficiency.

Evaluation & benchmarks

AI research agents with complex strategies can lose to simple searches

The authors introduce Auto-Research Arena, a multi-target benchmark for evaluating autonomous research systems in language modeling, spanning nine objectives that include model size, data, training, and inference efficiency. Across six LLM-based research harnesses and a Tree-structured Parzen Estimator baseline, a simple sequential search ranks second overall, indicating substantial room for improvement in current auto-research systems. The benchmark also surfaces a broad frontier of discoveries, some of which, when reused, yield better models under matched compute.