Evaluation & benchmarks

New benchmarks, evaluation methodology, measuring what models can do

456 articles · page 9 of 10

Evaluation & benchmarks

Long context windows do not ensure reliable sustained execution

Willette, Puvvada and Ginsburg introduce Long-Transduction, a diagnostic for whether models can repeatedly read state, transform it and emit correctly aligned outputs over long generations. Their experiments show that advertised context capacity does not imply dependable execution: accuracy fell substantially as context grew from 4K to 128K tokens, even though the underlying operations remained controlled.

Evaluation & benchmarks

Clinical agents falter when health records cannot support the question

Qiao et al. built an interactive benchmark from MIMIC-IV records to test clinical agents on paired questions that are either supported by the database or rendered unanswerable by a single evidence discrepancy. Across proprietary and large open-weight models, average task success fell from 62.2% on supported questions to 37.9% on noisy counterparts, showing that retrieval and query execution do not ensure evidence-grounded reasoning.

Evaluation & benchmarks

New benchmark disentangles sycophancy from empathy in multi-turn LLM conversations

The authors introduce FIGS, a benchmark with 500 multi-turn scenarios and an automated judge that separately scores sycophancy (factual yielding) and calibrated validation (appropriate empathy). Evaluating leading LLMs, they find a consistent trade-off: models gradually become sycophantic over sustained conversation, or shift to robotic dismissal, showing that balancing truthfulness with supportiveness remains unsolved.

Evaluation & benchmarks

Models struggle to predict software behavior after stateful interventions

Xinran Zhang introduces CTE-Bench, an executable benchmark that isolates whether models can predict the downstream effects of modifying a stateful software service. Across four API-hosted models, exact response accuracy on intervention-affected calls reached 54.3% to 61.5% when correct previous responses were supplied, but fell below 34% without that feedback or when models relied on their own predictions.

Evaluation & benchmarks

Coding agents struggle to turn software proposals into working changes

Peng et al. introduce LoLBench, a benchmark that asks coding agents to implement substantial software enhancements from human-written proposals rather than detailed coding specifications. The results separate two challenges: understanding how a high-level design maps onto an existing repository, and correctly implementing the resulting changes. Even the strongest evaluated agent resolved only 14 percent of tasks.

Evaluation & benchmarks

Adaptive testing exposes major security repair gaps in coding agents

The authors developed SecProbe, an evaluation framework that adaptively selects or synthesizes repository-scale vulnerability-repair tasks using Item Response Theory, a statistical method for matching test questions to ability. Across nine frontier models and two agent harnesses, performance remained low, while adaptive testing produced comparable ability estimates with fewer attempted tasks than baseline evaluation strategies.

Evaluation & benchmarks

LLMs often diagnose without correctly valuing available medical evidence

Feng et al. tested nine language models on MedEVM, where medical evidence arrives sequentially and models must choose whether to wait or diagnose. The authors find that models often submit diagnoses at the wrong evidential moment, remain sensitive to evidence order and misleading details, and gain 12.0 to 51.1 percentage points in accuracy when an evidence-verification harness controls submission.

Evaluation & benchmarks

Formal audit detects omitted source relationships in generative search

The authors specify a claim-gated audit procedure for generative search answers that checks whether a source relationship omitted from the answer is resolved. The procedure requires that relationship evidence, answer adoption, materiality, and disclosure are all observed; incomplete evidence remains unresolved. On an exhaustive synthetic suite, the reference checker reproduces all 81 three-state predicate combinations and rejects 192 deliberately malformed records.

Evaluation & benchmarks

Coding skill does not guarantee effective multi-agent software collaboration

The authors introduce a benchmark for evaluating collaboration among asynchronous coding agents using executable checks for dependencies between their assigned components. Across model families, scales, generations, and coordination protocols, stronger coding performance did not consistently produce better dependency resolution, while ordinary task metrics sometimes concealed coordination failures.

Evaluation & benchmarks

Agent skill for plugin migration shows reward gains but hides grading errors

The authors studied 64 reports on 16 static migration tasks, with two attempts per condition and 328 criterion decisions, evaluating a shipped plugin-upgrade skill. They found that the skill raised mean recorded reward by 4.92 points, but the gain was concentrated in one task, and eight task pairs were at the ceiling. Contract inspection and executable probes exposed grading errors that favored either arm, and re-grading with alternative judges gave gains of 10.63 and 6.09 points.

Evaluation & benchmarks

Retrieval-based checks miss many factual errors in open-ended medical answers

Huang et al. examine why retrieve-then-verify systems struggle to identify factual errors in long-form medical answers. Using an automatically induced error taxonomy, they find an order-of-magnitude performance collapse from closed-ended benchmarks to the clinician-annotated MedExpert dataset, with larger models, additional reasoning and broader evidence sources failing to resolve the problem.

Evaluation & benchmarks

Small-Sample LLM Rankings Often Overstate Evaluation Reproducibility

Sarkar repeatedly tested eight open model variants on prompt-structure inference, preserving 293 intermediate representations and auditing both output consistency and the resulting rankings. The models frequently produced different structures for identical calls, while bootstrap resampling and alternative merge rules showed that the evaluation identified the worst performers more reliably than the best.

Evaluation & benchmarks

Semantic alignment improves edge classification across unseen dynamic graph domains

Bonnet and Rei introduce a transfer-learning protocol for classifying future interactions in dynamic text-attributed graphs from previously unseen domains. Their experiments show that evaluated self-supervised graph methods underperform a simple text-only baseline on average, while their Spatio-Temporal Semantic Alignment model surpasses both that baseline and the existing methods.

Evaluation & benchmarks

AI tutoring matches expert human tutoring on GRE learning gains

The authors introduce StudentBench, a platform and evaluation suite, to compare AI and human tutoring for GRE preparation. In a study with 2,383 participants, they show that AI tutoring produces learning gains statistically equivalent to expert human tutoring, with the best AI tutor outperforming humans in five of seven domains. Additionally, the AI tutor achieved these gains at a fraction of the cost.

Evaluation & benchmarks

Checkpoint testing reveals hidden weaknesses in self-evolving agent memories

Lin et al. introduce EvoPathBench, which freezes an agent’s evolving memories or skills at successive checkpoints and tests each version on held-out episodes. Using public trading data, the authors find that endpoint scores conceal uneven development: gains often fail under distribution shift, some evolution paths suffer retention losses, and none of the tested methods adapts rules reliably.

Evaluation & benchmarks

Benchmark tests whether pretraining on multi-region neural recordings enables transfer across animals and tasks

The authors introduce BrainWideBench, a benchmark built on the International Brain Laboratory's Brainwide Map dataset (276 brain regions from 139 mice). They systematically evaluate pretraining methods for across-animal transfer and find that pretraining improves over baselines, but gains are heterogeneous and depend on alignment with downstream tasks. No method performs uniformly well across behavior, dynamics, and anatomy tasks.

Evaluation & benchmarks

Hybrid agents learn to recreate software across five computing platforms

The authors introduce reproducible environments for hybrid computer-use agents on Ubuntu, macOS, Windows, Android, and the Web. Agents recreate running applications without a prescribed workflow, while hidden behavioral and visual tests derived from the references provide automatic rewards; the strongest evaluated model scores 58.1% overall but fully passes programmatic tests on only 2.8% of tasks.