Evaluation & benchmarks

New benchmarks, evaluation methodology, measuring what models can do

456 articles · page 8 of 10

Evaluation & benchmarks

LLM persona prompts alter non-target attributes, breaking selective control assumptions

The authors test the assumption that changing one persona attribute in LLM user simulations changes only that attribute. They find a systematic cross-attribute influence across eight black-box and five open-weight models: e.g., describing a user as more risk-seeking shifts color choices, though color is never mentioned. Explicitly declaring a non-target attribute as neutral fails to stabilize it, while directional declarations succeed—a 'neutrality gap' that persona-following tests miss.

Evaluation & benchmarks

Multimodal models show a gap between aesthetic judgment and design action for UIs

The authors introduce AUV-Bench, a benchmark of 1,395 executable web UIs with four tasks: aesthetic scoring, diagnosis, repair, and text-to-UI generation. Evaluating 12 multimodal models, they find that while models show moderate agreement with professional designers on holistic scoring, exact diagnosis chain success peaks at 24.7%, and correct judgments do not consistently coincide with successful repairs, revealing a Judgment-Action Gap.

Evaluation & benchmarks

LLM agents complete tasks yet leak information across community boundaries, new benchmark shows

The authors introduce CoSec, an executable benchmark that tests whether persistent LLM agents respect privacy and authorization boundaries when operating in and across communities. Running 13 harness and model configurations on 208 scenarios, they find agents frequently complete legitimate tasks while violating those boundaries. Task utility, they show, does not imply security compliance.

Evaluation & benchmarks

Web agents struggle with multilingual, multimodal evidence hunts across the web

Aji and colleagues introduce HyperBrowseComp, a benchmark designed to test whether web agents can pursue implicit clues across languages and source types, rather than retrieve straightforward facts. The authors evaluated five web-enabled frontier models and found that the strongest configuration reached 31.68% accuracy, while most questions defeated every model using its native search system.

Evaluation & benchmarks

Codoku evaluates coding agents on program reasoning without execution

The authors introduce Codoku, a benchmark that presents coding agents with partial programs that cannot be executed, requiring them to fill typed cells to satisfy a specified control-flow graph and execution path. Because the puzzles are unexecutable, agents cannot use tools to bypass reasoning. Results from five frontier models show that even the best proprietary models solve only about half of the large puzzles, while small ones already challenge open-weight models.

Evaluation & benchmarks

Auditing passing trajectories exposes and repairs unearned benchmark passes

Luo et al. introduce a process-verification framework that audits agent passing trajectories, distinguishes evidenced reward hacking from verifier weakness, and localizes exploitable surfaces for repair. Across 3,810 passing trajectories from 29 model-benchmark cohorts, confirmed violations often increase with newer model generations but not monotonically. Three repair case studies show that blocking a recorded exploit is insufficient, and that combining minimal patches with exploit replay and fresh evaluation restores benchmark validity.

Evaluation & benchmarks

New dataset isolates and mitigates modality distraction in vision-language models

The authors introduce MoGround, a vision-language dataset where every question is guaranteed to be answerable from exactly one modality (vision or text). Measuring distraction across seven open-source VLMs, they find that which modality is more distractible depends on the model, and distraction scales inversely with the model's grounding strength in the relevant modality. A robustness vector trained on MoGround reduces distraction by 9–51% on all models with an average accuracy loss of only 0.1 points on standard multimodal tasks.

Evaluation & benchmarks

Coding agents fail at coordinated multi-repository programming tasks

The authors introduce WideSWE, a benchmark of 120 real-world software engineering tasks that require coordinated changes across multiple repositories. Evaluating seven coding agent configurations, they find that even the best performs poorly, with the Codex CLI–GPT-5.6-sol configuration achieving only 42.5% full task success. Trajectory analysis reveals agents frequently miss required repositories or stop after completing changes in only one.

Evaluation & benchmarks

New benchmark tests vision-language models under real-world visibility limits

The authors introduce SynDORBench, a benchmark of over 54,000 question-answer pairs that systematically degrade visibility according to the DORI standard (Detection, Observation, Recognition, Identification). Testing 16 open-source and one commercial large vision-language model (LVLM) plus a YOLO11x detector, they find that perceptual failure is governed by pixel density and imaging constraints rather than model scale. Several compact open-source LVLMs outperformed larger commercial baselines and YOLO11x under long-range and low-light conditions.

Evaluation & benchmarks

Benchmark reveals AI systems struggle to retrieve papers that inspire new research

Sohyeon Kim and colleagues built ScholarCatalyst, a benchmark where 184 lead authors of 207 computer science papers labeled which prior papers could have advanced their projects. The strongest system recovers only 48% of these catalyst papers in its top 20, and agentic search performs no better than embedding retrieval. This suggests that models lack the expert intuition needed to identify truly influential prior work.

Evaluation & benchmarks

Diagnosing where video models collapse diversity across dimensions

The authors introduce DiVid, a framework that profiles video generation diversity along six interpretable dimensions: Semantic, Style, Subject, Scene, Motion, and Camera. Systematic evaluation of representative models shows that even models with strong global diversity scores collapse on specific factors, particularly Motion and Camera, and that these failures stem from two bottlenecks: default-mode convergence under open-ended prompts and realization gaps when explicit alternatives are requested.

Evaluation & benchmarks

New benchmark PhysVista tests vision-language models on full physical cognition loop

The authors introduce PhysVista, a benchmark for evaluating physical intelligence in Vision-Language Models (VLMs). It is designed to assess the full seeing-reasoning-assessment loop, covering physical state perception, event- and scale-level reasoning, and physical plausibility assessment. Testing a range of VLMs reveals significant limitations in physical reasoning and plausibility judgment, highlighting a gap between visual recognition and true physical understanding.

Evaluation & benchmarks

Benchmark and method for true audio-visual joint reasoning

The authors introduce OmniReasoningBench, a benchmark of 1,150 questions where both audio and visual evidence are indispensable, and show that existing omni-modal benchmarks often allow single-modality solutions. They also propose a data engine (OmniQA) for generating training data with time-stamped clue chains, and a learning method (Modality-Factored Self-Distillation) that assigns token-level credit based on modality-specific contexts. Their model, OmniReasoning-30B-A3B, improves over the base model by 12.8 percentage points on OmniVideoBench and 9.3 points on OmniReasoningBench.

Evaluation & benchmarks

Chart editing benchmark separates completed requests from unwanted code changes

Tang et al. introduce a dataset and reference-free evaluation protocol for editing charts through their source code. ChartRevise contains 92,438 records across 344 edit types, and its evaluation distinguishes completed requirements, missed dependent updates and gratuitous changes rather than collapsing them into a single quality score. Fine-tuning improved exact-edit performance across several models and external benchmarks, although preserving unrelated content remains difficult.

Evaluation & benchmarks

LLM judges give inconsistent verdicts even at zero temperature

Ayyagari shows that pinning a judge to a fixed model version with temperature zero does not guarantee reproducible verdicts: identical inputs to four frontier judges on a major cloud platform yield per-item flip rates of roughly 5% on average and about 40% on close-call items that decide leaderboard margins. The aggregate ranking is stable, but roughly one-fifth to three-quarters of adjacent leaderboard positions become statistically indistinguishable, revealing a noise floor driven mostly by finite prompt sampling rather than the judge itself.

Evaluation & benchmarks

LLMs fail multi-step math despite solving each step correctly

Wang et al. introduce OracleLadder, a diagnostic tool that provides increasing levels of oracle help to large language models on multi-step math problems. By giving models a roadmap of sub-goals and their answers, they isolate failures into five categories. The dominant failure mode across all tested models is a composition gap: models can solve each intermediate step alone but cannot compose them into the correct final answer.

Evaluation & benchmarks

New benchmark tests automated theorem proving for research-level theoretical computer science

The authors introduce TCSAlgBench, a benchmark and reusable pipeline for natural-language proof discovery in theoretical computer science. It consists of 398 theorem-level challenges from 138 papers from STOC and COLT 2026. In their evaluations, the best-performing system (GPT-5.6 Sol max with discussion) achieved 23.6% verifier-accepted coverage after 10 rounds, while an agentic planning workflow reached 25.4%.

Evaluation & benchmarks

New benchmark tests if AI can track objects after they vanish from view

Ma et al. introduce Beyond3D, a VQA benchmark with 9,000 questions over 135 egocentric videos that tests whether VLMs can track an object's location after it is moved and has left the field of view. The best model achieved only 42.2% accuracy, with the largest failures in recovering when an object was last visible, indicating that current models struggle with out-of-sight spatiotemporal reasoning.

Evaluation & benchmarks

AI agents reproduce only 41% of machine learning papers with full code and weights provided

The authors introduce RECLAIM, a benchmark of 100 NeurIPS 2025 papers with predefined reproduction targets and GPU-hour budgets. They test four AI agents once per paper; the best agent reproduces 41% of Run-tier papers (code, data, weights provided), 27% of Retrain-tier (no weights), and 15% of Reimplement-tier (no code). Failed attempts use only 29% of their budget on average, and the most common error is implementing methods without verifying against the paper's reported numbers.

Evaluation & benchmarks

Video models struggle to connect cinematic techniques with storytelling intent

Xing and colleagues introduce CinematicVQA, a benchmark for reasoning about how cinematographic choices shape what viewers perceive and how a scene functions narratively. Their evaluations reveal a semantic gap: large vision-language models describe visible presentation more reliably than they identify the filming techniques behind it, while targeted fine-tuning improves narrative and multi-step reasoning.

Evaluation & benchmarks

Standard benchmarks overstate the accuracy of reused language model caches

Cestola et al. examine position-independent key-value cache reuse, a technique intended to accelerate retrieval-augmented generation by reusing cached representations of text chunks at different prompt positions. They find that common aggregate accuracy metrics and existing datasets can conceal the accuracy loss caused specifically by reuse, then introduce Boxoffice to generate more demanding evaluations.

Evaluation & benchmarks

New benchmark measures whether AI systems can explore, not just recall, unfamiliar rules

The authors introduce ExplorationBench, a benchmark that evaluates AI systems' ability to explore unknown environments by forming hypotheses, designing experiments, and iterating on results. Its two executable sandboxes make every answer exactly checkable while ensuring that pretraining recall alone cannot solve the tasks. Evaluating ten AI systems, the authors find that the strongest can acquire and apply these unfamiliar rules, though performance varies across trajectories and continued exploration can stall or reverse earlier gains.

Evaluation & benchmarks

Clinical language models diagnose well but often mishandle full encounters

Fang et al. introduce a clinician-authored benchmark that evaluates the process of clinical assessment rather than diagnosis from a complete case summary. Across 31 models, diagnostic accuracy reached 90.7%, yet even the strongest systems passed fewer than 30% of tasks when required to question a virtual patient, request appropriate examinations, respect constraints and provide the correct diagnosis.

Evaluation & benchmarks

Security agents lose evidence support when available telemetry changes

The authors introduce APTInvestBench, a benchmark that tests whether LLM agents can investigate advanced persistent threats when the available security logs change. Across eleven models, agents retrieved sufficient evidence for 44.3% of recoverable attack actions on average, but their final citations adequately supported only 25.0%, revealing substantial losses between finding evidence and reporting it.