Evaluation & benchmarks

New benchmarks, evaluation methodology, measuring what models can do

456 articles · page 4 of 10

Evaluation & benchmarks

Systematic audit reveals pervasive data quality issues in RVL-CDIP document classification benchmark

Larson and Leach systematically audited the 400,000-document RVL-CDIP benchmark and found multiple defects: label errors, train-test leakage, spurious cues, OCR sensitivity, and sensitive personal data. After remediating all defects, retrained models were 0.9 to 2.9 points more accurate; the most accurate model, LayoutLMv3, reached only 96.5%, far below the 98.8% claimed on the original benchmark.

Evaluation & benchmarks

Short-horizon PDE model accuracy does not ensure long-rollout reliability

The authors evaluate seven PDE models on seven time-dependent datasets, comparing short-horizon and full-horizon autoregressive rollouts. They find that short-horizon advantages of foundation models often preserve rankings but absolute errors grow substantially, with median error amplification of 2.04 across 144 configurations. Over 40% of configurations that fall below a VRMSE of 1 at short horizon exceed it at full horizon, and 12 exceed VRMSE 10 despite none doing so locally.

Evaluation & benchmarks

AI annotators better reflect human labeling disagreements after learning from feedback

The authors construct ANT-A, a multi-annotator benchmark spanning 7 datasets across domains and modalities, each with detailed guidelines and 5 independent human labels per instance. They propose ANT-OR, a multi-agent annotation framework that refines prompts by distilling contrastive decision rules from positive and negative feedback. ANT-OR improves agreement@>=2 accuracy by 4.5 points over the strongest baseline and achieves Fleiss kappa within 2 points of human annotator distributions.

Evaluation & benchmarks

AI models underuse sound when interpreting ambiguous text and images

The authors introduce TRIAD, a benchmark of 327 text–image–audio queries each designed so that the correct answer is unique only when all three modalities are present. Evaluating 14 omni-modal systems, they find that the best model achieves 71.2% group accuracy, falling over 20 points behind human performance, and that removing multiple-choice options drives model scores into the single digits.

Evaluation & benchmarks

Prompting strategy effects depend on judge model and language in LLM evaluation

Mehrabi et al. evaluate 9 prompting strategies on 4 open-weight LLM judges and 27 languages across 4 tasks, collecting over 1 million pointwise scores. They find that the judge model explains far more score variance than the prompting strategy, and that strategy effects often reverse direction across judges, especially for low-resource languages where aggregate metrics conceal these differences.

Evaluation & benchmarks

Dual Exam benchmarks agents and world models separately, exposing weak coupled performance

The authors introduce Dual Exam, a two-track evaluation framework that tests multimodal agents under fixed environment responses and world models under fixed action interventions across physical, digital, social, and scientific domains. On 1,088 agent items and 400 world model cases, top scores were 31.3 for GPT 6 Astra and 21.7 for GPT-Image-2, while a 150-case coupled pilot achieved zero task successes.

Evaluation & benchmarks

Reinforcement learning yields user simulators that stay faithful and sound natural

The authors address a trade-off in user simulators for evaluating AI assistants: prompted LLMs follow complex intents but sound unnatural, while specialized simulators sound human but drift from task instructions. They propose a multi-turn reinforcement learning recipe that jointly rewards task intent coverage and behavioral naturalness, using a dynamically retrained discriminator to judge naturalness. The resulting model, UserLM-2.0, achieves valid conversations 46.2% of the time versus 14.1% for the strongest prior simulator.

Evaluation & benchmarks

AI agents struggle more with understanding what they see than reasoning

The authors introduce AgentVQA, a curated offline benchmark of 1,000 multiple-choice questions drawn from 23 datasets across five agentic domains: web agents, robotics, egocentric video, games, and spatial planning. Evaluating 22 vision-language models, they find the best reaches only 58.5% accuracy, and error analysis attributes most failures to inadequate grounding and tracking of visual state rather than to task misunderstanding or reasoning errors.

Evaluation & benchmarks

Cheap baselines often rival expensive LLM methods on graph node classification

Podstawski tunes four cheap baseline models on splits from three recent studies of LLM-based node classification on text-attributed graphs, comparing them to the best reported LLM accuracy. Correct & Smooth on TF-IDF features trails the reference by a median of 2.3 points and an MLP on frozen embeddings by 2.8 points. The authors also document several evaluation risks in released code, including test-set hyperparameter selection and label leakage.

Evaluation & benchmarks

First benchmark evaluates coding agents on XR development tasks

The authors introduce GenXR-Bench, the first benchmark for end-to-end evaluation of coding agents in extended reality (XR) software development. It comprises 112 tasks spanning SDK engineering and application generation. Evaluating 10 model configurations, the strongest achieves 50.9% overall Pass@1; failure analysis reveals recurring misunderstandings in coordinate transformations, asynchronous sensor lifecycles, and bimanual interaction semantics.

Evaluation & benchmarks

Simulated users test coding agents as requirements unfold across conversations

Brehme and colleagues introduce SWE-DIVE, a framework for evaluating coding agents through conversations rather than a single complete task description. In a preliminary evaluation across 11 tasks, the authors report that simulated user personas produce distinct interaction behaviors and that unconstrained or changing requirements lead to more code regressions, changes that break previously working behavior.

Evaluation & benchmarks

Benchmark tests proactive memory for wearable AI assistants in streaming egocentric video

The authors introduce ProactEgoMem, a benchmark for proactive memory in streaming egocentric video. It defines three object-centric tasks, Information Capture, State Tracking, and Location Tracking, at two levels of proactiveness (notification and intervention). Evaluating multiple proprietary, open-source, video-language, and streaming baselines shows that even the strongest models fail to capture memory both timely and accurately.

Evaluation & benchmarks

Tree-based method reveals fine-grained LLM capabilities hidden by aggregate scores

The authors propose TIDE, a method that induces a decision tree from benchmark samples and model scores to diagnose fine-grained capabilities. Applying TIDE to five models across four benchmarks, they find that aggregate scores conceal substantial variation: for example, Claude Opus 4.6 is perfect on microeconomic reasoning while GPT-5.4 scores 83.3%, but GPT-5.4 outperforms on contract law reasoning (88.9% vs 77.8%).

Evaluation & benchmarks

Benchmark tests LLM agents on recreating published computer science contributions

The authors introduce FrontierCS 2, a benchmark of 48 open-ended tasks across 13 computer science domains. An agent inherits a repository with a paper's contribution deleted and must submit patches to beat the original code on every metric and workload, judged privately. The best agent, Claude Opus 5.5, passed 58% of runs, but no agent passed any of 16 tasks, and most high-scoring runs still failed on some metric.

Evaluation & benchmarks

Coding agents struggle with system-level performance optimization in microservice systems

Zhang et al. introduce MicroPerf-Bench, a benchmark for evaluating coding agents on end-to-end performance optimization in microservice systems. They injected empirically grounded performance anti-patterns into seven real-world open-source systems, creating 65 optimization instances. Evaluating nine agent-model configurations over 1,755 runs, the best configuration achieved a normalized score of only 42.0%, with the dominant failure being functionally correct changes that failed to improve performance.

Evaluation & benchmarks

Compaction tuned for one query fails to aid later queries

The authors evaluate context compaction methods on later, unseen questions across four multi-query corpora. They find that query-aware sentence selection, which keeps sentences relevant to the current question, provides no measurable improvement over random deletion for subsequent questions. Meanwhile, word-level compressors such as LLMLingua-2 achieve significant gains on later questions by retaining more content words.

Evaluation & benchmarks

RAG's retrieval, not reading, is the bottleneck for long-context question answering

The authors propose evidence-based generation (EBG), a task in which models must return an answer with line-level citations, and a joint Ans+Evid metric that scores correctness and grounding simultaneously. Testing ten proprietary frontier and three open-weight LLMs on four evidence-grounded benchmarks spanning 10K to 1M characters, they find that RAG's similarity retrieval misses required chunks for multi-fact queries and that chunking discards identity-critical document metadata. Long-context models avoid both failure modes.

Evaluation & benchmarks

New benchmark reveals agents fail to detect and recover from their own errors in long tasks

Hu et al. introduce CUA-Recovery, a benchmark for evaluating error recovery in computer-use agents operating over hundreds of steps across multiple applications. The authors constructed a test suite of 2,250 human-verified erroneous states drawn from failures of six different agents, and a training suite of 8,600 verified recovery trajectories. Their evaluation shows that current agents rarely detect errors inherited from a prior policy and rarely complete the task, while fine-tuning on the training data substantially improves both error awareness and task completion.

Evaluation & benchmarks

Benchmark shows AI agents struggle with fine-grained 3D spatial coherence in Blender

The authors introduce BlenderWorld, a benchmark of 223 realistic 3D creation tasks in Blender, spanning reconstruction, modification, and design from text, image, video, and structured specification inputs. They evaluate 13 frontier multimodal agents using 2,651 task-specific binary rubrics alongside holistic visual quality scores, finding that the best model, GPT-6-Astra, reaches an overall fused score of 79.9 yet passes every requirement on just 42.2% of tasks.

Evaluation & benchmarks

Accessibility profiles reveal hidden tradeoffs in LLM reliability

The authors introduce accessibility profiles that evaluate language models across multiple meaning-preserving formulations of each inquiry. They find that for about half of model pairs with nearly equal mean accessibility, which model looks better depends on whether one success in five formulations is enough (Access@5) or all five must succeed (Reliable@5), with gaps reaching 10 percentage points in opposite directions.

Evaluation & benchmarks

Safety benchmark scores reflect training policy more than model capability

Jebraeeli et al. exploit an industrial accident where roughly 53,000 prompts were labeled under two different safety policies (moderator block vs. harmful content). Training ten guardrail models at two scales, they find that benchmark scores are heavily influenced by the labeling policy: HarmBench-copyright varies by 0.99 across arms, and over-refusal benchmarks vary by 53 percentage points. They also show that training corpora often contain benchmark prompts (e.g., 88% of JailbreakBench-benign is in training data), and validation accuracy tracks policy agreement (rho=+0.74) rather than detection capability (rho=+0.34).

Evaluation & benchmarks

Agent memory narrows LLM response diversity, limiting user access to information

Tan et al. introduce MemBubble, a benchmark of 1000 query-history pairs built from real user-assistant conversations, to measure how agent memory affects response diversity. Across six LLMs and five memory mechanisms, they find that memory processing reduces response entropy by 34.7% on average compared with providing the full interaction history. Simple user-side interventions fail to close this diversity gap.

Evaluation & benchmarks

State media articles now dominate English web crawl content

The authors introduce WorldStateMedia, an open dataset of 118 million state media articles in 114 languages, built by linking Fineweb web crawl data to outlet-level state control annotations. Auditing these corpora, they find editorially controlled state media has grown sharply in English crawls and is overrepresented in low-resource languages. A retrieval-augmented generation experiment shows that retrieving country-specific state media biases LLM answers toward that country's territorial claims.