Evaluation & benchmarks

New benchmarks, evaluation methodology, measuring what models can do

97 articles · page 1 of 2

All Agents Language models Reinforcement learning Vision Generative media Speech & audio Robotics & embodied AI Safety & security Evaluation & benchmarks Efficiency & systems Code & software AI for science & health Theory

Viewgrid list compact

LLMs fail multi-step math despite solving each step correctly

LLM judges often encode better preferences than their verdicts reveal

New benchmark tests automated theorem proving for research-level theoretical computer science

Video-language models often bind actions to the wrong people

New benchmark tests if AI can track objects after they vanish from view

Benchmark exposes distinct weaknesses in multimodal audio-video generation systems

Surrogate benchmarks enable systematic evaluation of scaling analysis methods across model types

Indic-centric corpus of 1.16 million sentence pairs improves machine translation for Indian languages

AI agents reproduce only 41% of machine learning papers with full code and weights provided

Video models struggle to connect cinematic techniques with storytelling intent

LLMs struggle to match multidisciplinary tumor board discussions in new benchmark

Standard benchmarks overstate the accuracy of reused language model caches

New benchmark measures whether AI systems can explore, not just recall, unfamiliar rules

Coding agents nearly finish repositories but still miss explicit requirements

Voice agents lose document grounding as conversations and context grow

Top AI agents complete fewer than one-quarter of professional tasks

Agent performance depends more on system configuration than model size

Benchmark exposes gap between choosing tools and using them

Language agents rarely withdraw selectively when instructions or permissions change

Clinical language models diagnose well but often mishandle full encounters

Self-updating agents can improve performance while quietly weakening safety

Shopping agents falter when user needs change across a conversation

Security agents lose evidence support when available telemetry changes

Frontier vision-language models mostly fail a real low-speed driving course

Long context windows do not ensure reliable sustained execution

Correcting search bias makes deep research conclusions more representative

Clinical agents falter when health records cannot support the question

New benchmark disentangles sycophancy from empathy in multi-turn LLM conversations

Sparse Human Label Overlap Undermines Reliable LLM Judge Validation

Fixed LLM judges can mismeasure upgrades as agents change

Models struggle to predict software behavior after stateful interventions

Coding agents struggle to turn software proposals into working changes

Conversational graph querying exposes a gap in model autonomy

Correct Answers Often Fail to Prove Genuine Multi-Hop Reasoning

LLM agents deploy repairs better than systems built from scratch

Generating user guides can improve web agents’ action selection

Agent defenses leave privileges unseen by standard security benchmarks

Marketplace Steering Often Diverts Computer Agents From Users’ Best Choices

Fixed controls expose limits in financial LLM agent evaluation

Email agents execute valid tool calls but often miss the task

Sparse annotation overlap makes LLM judge validation decisions unreliable

Adaptive testing exposes major security repair gaps in coding agents

Search agents struggle to find broad evidence for open research problems

Task-aware certificates safely automate part of LLM agent evaluation

Benchmark tests agent control of anomaly detectors under delayed feedback

LLMs often diagnose without correctly valuing available medical evidence

GUI agents sacrifice future options in long visual puzzles

Faulty harnesses can radically distort prompt injection security results

Formal audit detects omitted source relationships in generative search

Coding skill does not guarantee effective multi-agent software collaboration

Current LLM agents struggle to sustain long-horizon team collaboration

False accusations can make LLM agents sabotage correct work

Agent skill for plugin migration shows reward gains but hides grading errors

Compiler turns scientific protocols into certified simulated laboratory tasks

Web agents struggle to turn search results into usable artifacts

Retrieval-based checks miss many factual errors in open-ended medical answers

Small-Sample LLM Rankings Often Overstate Evaluation Reproducibility

Structured evidence reports reduce false success after tool failures

Semantic alignment improves edge classification across unseen dynamic graph domains

Goal pressure exposes wide gaps in security agents’ scope adherence

Replayable news timelines test how LLM agents update forecasts

Unverified state text can flip calibrated model decisions

Competitive games provide a dynamic benchmark for language model strategy

Benchmark tests whether coding agents improve governance without breaking repositories

Enterprise agents struggle when facts are hidden across business systems

Multi-image evidence can help models repair software, but unreliably

AI tutoring matches expert human tutoring on GRE learning gains

Checkpoint testing reveals hidden weaknesses in self-evolving agent memories

Process-based evaluation reveals where computer-use agents go wrong

DolphinBench measures agent memory through tasks, cost, and latency

Benchmark tests whether pretraining on multi-region neural recordings enables transfer across animals and tasks

Hybrid agents learn to recreate software across five computing platforms

Voice-agent models diverge most beyond choosing the correct tool

Coding agents often overstate how thoroughly they reviewed files

Deep research agents struggle with long, multimodal evidence chains

Refresh schedules, not cache age, determine agent answer staleness

Cross-device workflows expose major weaknesses in today’s GUI agents

API benchmark scores poorly predict performance in consumer chatbot interfaces

Benchmark tests retrieval for agent-written queries across 190 million web pages

Current robot models falter as spatial and procedural complexity rises

LLM judges as measurement instruments fail reliability tests on shared endpoints

Coding agent benchmarks miss review compliance gaps

Civilization benchmark exposes agents’ weak monitoring and long-term follow-through

Embedding retrieval ranks matching words above shared underlying structure

LLM agents do not reliably improve from their own experience

Models struggle to improve themselves from vague capability goals

Models remain weak at steering coding agents through long tasks

LLMs fail at multi-round code review in new realistic benchmark

Process-level tests expose distinct mathematical agent capabilities in LLMs

MLLMs fail to sustain goal-directed navigation in real-scale 3D city

LLM agents struggle to infer wellbeing from long-term wearable data

Domain context improves LLM agents’ time-series root-cause attribution

Bayesian framework exposes hidden differences among seemingly similar RAG systems

Financial agents can cite rules while still attempting prohibited trades

New benchmark standardizes evaluation of coding agent harnesses

New benchmark reveals coordination as distinct bottleneck for LLM agents