Evaluation & benchmarks

New benchmarks, evaluation methodology, measuring what models can do

456 articles · page 5 of 10

Evaluation & benchmarks

Utility metric reveals that accuracy doesn't guarantee responsible AI decisions

The authors present UtilityBench, a 15,000-instance benchmark that assigns sample-specific costs to errors using Best-Worst Scaling, and UtilityAUC, a metric that measures how well model confidence aligns with the utility of outcomes. Evaluating 20 state-of-the-art language models, they find that greater accuracy and more advanced architectures do not reliably translate to higher UtilityAUC, revealing a gap between problem-solving ability and responsible decision-making.

·2 min
Evaluation & benchmarks

An agentic framework that diagnoses and verifiably repairs foundation model failures

The authors introduce EvalRx, an open-source agentic framework that discovers systematic failure modes in foundation models, formulates falsifiable hypotheses, and verifies them before applying targeted repairs. In evaluations across 12 model family-modality combinations spanning Gemini, Qwen, Gemma, and Nemotron, EvalRx improved performance by +1.4 to +10.9 percentage points, with an average gain of +5.8 points.

·3 min
Evaluation & benchmarks

Southeast Asian language models still rely heavily on English representations

Han and colleagues introduce DeepSEA, a professionally translated benchmark covering mathematical and scientific reasoning, question answering, summarization and machine translation in 12 Southeast Asian languages. Their evaluation finds that leading frontier systems remain comparatively robust, while smaller models deteriorate sharply on low-resource languages and often perform better when using English as an intermediate language.

·3 min
Evaluation & benchmarks

Benchmark reveals long-context models falter on state tracking and pattern learning

The authors introduce LEAPS, a collection of 12 novel datasets designed to evaluate long-context language models on state tracking and pattern learning, capabilities often overlooked by existing recall-focused benchmarks. Testing 30 API and open models at context lengths from 8K to 512K tokens, they find that models scoring near-perfect on recall tasks vary dramatically on these other abilities and degrade significantly as input length increases.

·2 min
Evaluation & benchmarks

New benchmark quantifies physical plausibility in AI-generated videos

Kaiwei Chen and colleagues introduce PVBench, a benchmark that systematically evaluates physical plausibility in text-to-video and image-to-video generation. By pairing 600 prompts with vision models that extract object-level physical quantities—such as 6D pose, metric scale, and kinematic residuals—the authors show that existing models score significantly lower on physical plausibility than on semantic quality, highlighting a persistent gap between visual realism and physical coherence.

·3 min
Evaluation & benchmarks

Even top forecasters fail when future events set covariates to unseen values

The authors introduce WhatIf-Bench, a semi-synthetic benchmark of 37,665 tasks built from mechanistic simulators of data centers, power grids, road traffic, and water networks. It measures how well forecasters use covariates to predict a target's response to future events. The central finding: every model forecasts well when events replay seen covariate values, but skill drops sharply when events set covariates to values never seen in history.

·3 min
Evaluation & benchmarks

AfriGuard safety dataset for 10 African languages shows guardrails lag general LLMs

The authors introduce AfriGuard, a culturally grounded multilingual dataset covering safe and unsafe prompts, harm categories, and refusal behavior in 10 African languages plus English. They find that specialized guardrail models underperform general-purpose LLMs on safety classification, and that models often fail to refuse unsafe requests because they produce linguistically inadequate responses rather than explicit refusals. Fine-tuned models built on AfriGuard match models up to 30 times larger and improve refusal detection.

·2 min
Evaluation & benchmarks

Benchmark tests AI agents' ability to jointly improve training data, trainer, and inference harness

Peng et al. introduce VibeTraining-Bench, a benchmark in which an agent must iteratively refine a target system's training data, trainer, and inference harness within a fixed resource budget. Across terminal operation, software engineering, multi-turn tool use, and mathematical reasoning, frontier research agents improved target systems, approaching human-developed baselines on SWE-bench Verified. However, the authors find that agents rarely venture beyond lightweight data filtering and hyperparameter tuning.

·1 min
Evaluation & benchmarks

Frontier text-to-image models score as low as 23.9 on hardest benchmark

The authors introduce GenOlympiad, a benchmark of 801 deliberately difficult text-to-image prompts spanning nine capability axes, with an automated pipeline that decomposes each prompt into atomic, visually verifiable questions. Evaluating more than 20 frontier models, they find that the best text-to-image model, Nano Banana Pro, achieves only a strict score of 23.9 out of 100, while a coding-focused model achieves higher prompt alignment but noticeably worse aesthetics.

·3 min
Evaluation & benchmarks

Benchmark reveals AI agents struggle with visual-reference web development

The authors introduce RefCraft, a benchmark of 300 realistic web development tasks that combine natural language instructions with screenshots from one to five reference webpages. Evaluating nine frontier multimodal models in an inspect-and-iterate setting, they find that even the best system, GPT-6 Astra, achieves only 52.12 out of 100, indicating that visual-reference-conditioned agentic generation remains a major challenge.

·2 min
Evaluation & benchmarks

New benchmark shows LLM routers ignore latency when feedback is incomplete

The authors introduce RouterWorld, a benchmark that reunifies 371k single-turn responses and 76k multi-turn agentic trajectories across seven open-weight LLMs, and formalizes partial observability via Cold-Start and Voyager setups. Evaluating 13 routing baselines, they find that under realistic partial observability, most routers over-emphasize accuracy at the expense of latency, with a worst-case router collapsing into a degenerate policy.

·3 min
Evaluation & benchmarks

Proactive assistance requires choosing the right moment, not just the right words

The authors introduce ProPerMem, a benchmark for memory-driven proactive assistance that pairs egocentric video with year-long, distractor-rich personal memories. Across all baselines they evaluate, strict accuracy is at most 15%. A controlled factorial study shows content and timing are separately necessary: removing either collapses performance from a 96% oracle ceiling to 38% and 15%.

·2 min
Evaluation & benchmarks

Text-only finetuning improves attribute-object binding in vision-language models across all retrieval pairings.

The authors introduce SABER, a synthetic benchmark that varies attribute-object assignments while holding the object and attribute sets fixed, and evaluate retrieval across all four image-text modality pairings. They find substantial binding failures in CLIP-style dual encoders and MLLM-based embedders, then show that finetuning an MLLM-based embedder on synthetic text alone, with no image supervision, improves binding across all pairings and sets a new state of the art on SABER.

·2 min
Evaluation & benchmarks

Formal plan validation shows LLM agents rarely plan as claimed

Miro-Lopez-Feliu et al. test whether LLM-based agents can actually plan by grounding natural language tasks in hidden PDDL models that mechanically validate every action and goal. Across four models, six prompting strategies and three agent architectures, the best offline configuration solved 78% of solvable tasks, but online success fell to 40%, and unsolvable tasks were recognized in under 10% of episodes.

·2 min
Evaluation & benchmarks

LLM agents can repair causal graphs beyond algorithm selection

Ma, Ding and Wang introduce Heuresis, a benchmark designed to test whether LLM agents can improve causal graphs through targeted analysis rather than prior knowledge or simple algorithm selection. On six families of simulated data, GPT-6.1 Sol consistently exceeded the best selected conventional algorithm, while trace analysis showed that some recovered edges required evidence not present in any baseline algorithm’s output.

·3 min
Evaluation & benchmarks

Mobile agents are broadly vulnerable to cross-modal injection attacks that hijack reasoning

The authors introduce MobileInject, the first benchmark for evaluating multimodal large language model based mobile agents against indirect prompt injection attacks. Across 11 realistic scenarios, 9 security-risk categories, and 108 executable risk instances, they find that mobile agents are broadly vulnerable, with many failures traceable to reasoning hijacking in early stages.

·2 min
Evaluation & benchmarks

Benchmark exposes when AI reasoning ignores audio-visual evidence

The authors introduce ANCHOR, a benchmark of 2,319 videos and 8,272 question-answer pairs with structured reasoning traces, temporal evidence, bounding boxes and audio cues. Evaluating twelve multimodal large language models, they find that comparable final-answer accuracy does not imply grounded reasoning; removing video input drops accuracy by 38.4 to 41.8 points, while audio and bounding box removals cause smaller, model-dependent declines.

·3 min
Evaluation & benchmarks

Typed decision models show bias but offer cheaper, consistent judgment

Lukassen, Weisser, and Silbersdorff systematically tested eight typed decision models (TDMs) on the CALM bias benchmark for LLM judges. They found that every TDM is biased in three to eight of twelve tests, with emotional tone and position affecting all models. Despite being more biased than 2026 LLM judges, TDMs are cheaper and more consistent, making them a viable option for pairwise judgments if biases are evaluated.

·2 min
Evaluation & benchmarks

Neither decision models nor new LLMs beat statistical baselines as synthetic survey respondents

Lukauskas and Šarkauskaitė tested 14 decision models and 47 generative LLMs as substitutes for human survey respondents. The best generative model, MiniMax M3, reached a Psychometric Similarity Score of 0.813, matching a Gaussian copula fit only on other respondents and falling below an in-sample copula fit. Neither model class outperforms simple statistical baselines.

·3 min
Evaluation & benchmarks

Executability-oriented benchmark exposes LLM service-call grounding failures

Wang et al. introduce SCCGBench, a benchmark of 3,135 real-world RapidAPI call functions mined from GitHub, designed to test whether LLM-generated code actually invokes the correct service endpoint. They evaluate six code LLMs and find that models often generate plausible-looking but non-executable code, with the best model achieving only 3.76% Mock success (hitting the correct host, endpoint, and HTTP method) without documentation. Providing gold API documentation boosts Mock to roughly 90%, highlighting that API grounding, not code generation, is the primary bottleneck.

·3 min
Evaluation & benchmarks

LLM agents degrade reliably under controlled interaction perturbations

The authors introduce AgentPerturb, a framework that applies missing, redundant, and incorrect edits to user utterances, procedural specifications, carried-over context, and tool contracts in agentic tasks. Evaluating ten models on APEX-Agents and Toolathlon, they find consistent performance drops, with vulnerability organized more by perturbation target than type, and successful perturbed trajectories often requiring substantially more computation.

·2 min
Evaluation & benchmarks

Outcome verification reveals failures in social agent plans that conversation ratings miss

Kang et al. introduce HICO-Bench, a benchmark of 180 everyday coordination scenarios where two agents with private information must agree on a plan. Each scenario has two versions differing in one private condition, and a program checks whether the plan is feasible (matches records and satisfies conditions) and an LLM judge checks integrity (conversation supports the plan). Across thirteen LLMs, 36% of conversations rated highly for goal completion fail these outcome checks. The authors also propose Paired-Credit Policy Optimization (PaCPO), a training method that uses tentative plans at each turn across the paired versions to assign credit, raising Qwen3-8B's verified success from 20.4% to 28.3%.

·3 min
Evaluation & benchmarks

Transcripts from saturated benchmarks still predict model capability

Hamada et al. show that once a benchmark's scores saturate, the transcripts of how models solved the tasks retain information about their capabilities on harder, held-out items. They simulate saturation retrospectively and compare four transcript-based protocols, finding that pairwise LLM judging extracts the strongest signal, cutting prediction error by 5.3 times compared to a baseline using only saturated scores. The capability signal is spread across transcript sections, but certain channels, such as reasoning and tool calls, carry more weight depending on the domain.

·2 min
Evaluation & benchmarks

Semantic reconstruction game makes jailbreak evaluation more reliable across judges

The authors introduce a semantic reconstruction game that evaluates jailbreak success by inferring what request a response actually answers, then comparing that reconstruction against decoy requests. Across three off-the-shelf LLM judges, it achieves the lowest mean error against human labels among compared evaluators. Their method, which deliberately avoids false positives, substantially stabilizes reported attack success rates.

·2 min
Evaluation & benchmarks

Video AI models trust misleading on-screen claims over what the footage shows

The authors introduce VisualTrust, a counterfactual benchmark of 5,063 video-question cases in which the same clip appears with no claim, an accurate claim, or a misleading claim. Across 11 multimodal large language models, accuracy drops from 90.1% with no claim to 51.9% with a misleading claim, and most errors are directed flips to the answer the claim supports. The authors also propose DEAR, a training-free agent that separates evidence arbitration from reading and largely restores accuracy.

·2 min
Evaluation & benchmarks

Video generation models often fail to generate causally coherent physical event chains

The authors introduce CausalCascade, a simulator-grounded benchmark of 50 Blender scenes with structured event graphs and 2,832 controlled counterfactual variants. Evaluating nine image-to-video models, they find that while more detailed future-event specification improves event recovery, the faithful physical realization of causal cascades remains limited, and models show weak selective responsiveness to physical interventions.

·3 min
Evaluation & benchmarks

LLMs can say no but still execute harmful tool calls, with language-dependent gaps

The authors introduce Lumen-bench, a behavioral benchmark for tool-calling ethics spanning 30 models, five languages (Portuguese, English, Spanish, Mandarin Chinese, Hindi), six social-pressure variants, and two system prompts, totaling 194,400 trials. They find that harmful tool calls can co-occur with verbal refusals (a "say-do" gap estimated at 8.57%), that language differences persist even within the same model (e.g., higher harmful-action rates in Hindi and Portuguese than English under an ethical prompt), and that replacing an ethical prompt with a neutral one raises pooled violation rates from 2.76% to 13.64%.

·3 min
Evaluation & benchmarks

SkillHorizon benchmark reveals that language agents struggle to complete long business workflows as task scope grows

The authors introduce SkillHorizon, a benchmark that constructs long-horizon agent tasks by composing reusable skills across seven business environments. They evaluate five language models on 1,050 tasks at various scales. The key finding is that as reference interaction workloads increase, agents complete a decreasing proportion of local goals, with mean completion dropping from 90.4% to 46.8%.

·2 min
Evaluation & benchmarks

Survival sandbox benchmark evaluates autonomous agents on sustained goal execution

The authors introduce ASIWorld, a survival sandbox with persistent world and character state, character-scoped observations, and a programmable asynchronous interface. They evaluated nine agent configurations on two tasks, production and exploration. Eight configurations completed the production task and six completed both tasks, but analysis of execution trajectories revealed significant differences in preparation, combat, and recovery behaviors that endpoint success alone did not capture.

·2 min
Evaluation & benchmarks

AI agents still cannot reliably deliver end-to-end on underspecified business requests.

The authors present FDE-Bench, a benchmark of 49 cases built from real customer projects. Agents must clarify underspecified requirements with a simulated customer, then iteratively optimize a solution until it meets acceptance criteria. Across 18 agent configurations, the best mean delivery score was 0.5510 against a customer-accepted reference of 1.0, and the best delivery success rate was 6.12%.

·2 min
Evaluation & benchmarks

Diagnostic feedback drives iterative benchmark evolution revealing hidden LLM agent weaknesses

The authors propose EvoProbe, a framework that uses diagnostic feedback from LLM agent evaluations to iteratively generate new, more challenging tasks. In cybersecurity experiments, all eight models tested on 66 tasks showed decreased performance on updated tasks, with mean Outcome scores dropping 11.8% and mean Pass^3 rates falling 40.0%. The framework reveals phase-level failure patterns, such as wrong hypothesis and insufficient exploration, that aggregate scores obscure.

·3 min
Evaluation & benchmarks

AI agents fail to produce honest scientific follow-up papers in new benchmark

The authors introduce Into the Unknown, a benchmark that tasks AI agents with writing a follow-up scientific paper given a research direction from an unpublished study. Across 135 submissions from nine frontier agents, none met all three criteria: integrity (no fabrication), completeness, and sufficiency as judged by domain scientists. 114 submissions failed the integrity check, most often due to claims of work that was never done or numerical contradictions with recorded evidence.

·3 min