Evaluation & benchmarks

New benchmarks, evaluation methodology, measuring what models can do

456 articles · page 10 of 10

Evaluation & benchmarks

LLM judges as measurement instruments fail reliability tests on shared endpoints

Haoyuan Zhu and Jie Zhang preregistered two evaluation campaigns that assumed LLM judges (used to score generations, filter training data, or rank leaderboards) would produce stable outputs for identical inputs on shared endpoints. They found that the instruments themselves failed validation: same-window repeat rankings reached only Spearman 0.400 against a 0.90 threshold, and byte-identical next-day replays reached 0.78 against 0.99. The authors document three mechanisms—a label-to-meaning mapping bias, candidate gaps below the noise floor, and input-identical ranking noise—and show that neither metric substitution nor sampling fixes the problem.

·3 min
Evaluation & benchmarks

Embedding retrieval ranks matching words above shared underlying structure

Rashid and Kellis tested structural retrieval under controlled surface-form changes using 500 mathematics queries and 118 embodied-agent queries. At the hardest mathematics tier, two production embedding models scored 0% strict Hit@1 despite placing the correct item in the top 10 nearly every time; trajectory retrieval fell to chance or below when matching required different object names. Reranking with an LLM improved results, but some mathematics gains appeared attributable to competition-problem memorization.

·3 min
Evaluation & benchmarks

LLM agents do not reliably improve from their own experience

Shi and colleagues introduce S³Gym, a benchmark that separates open exploration from held-out evaluation to test whether language-model agents can examine, assess, and learn from their own behavior. They find that experience can improve later decisions, but gains vary substantially by model, game, and whether experience is retained as raw history, compressed into summaries, or incorporated through parameter training.

·2 min
Evaluation & benchmarks

MLLMs fail to sustain goal-directed navigation in real-scale 3D city

The authors introduce UrbanGround, a sandbox built from 3D geospatial data of Hong Kong, to test whether multimodal large language models (MLLMs) can turn local visual perception into sustained navigation. They find that while agents show useful abilities in short-range spatial reasoning and scene recognition, they fail at orientation, pedestrian-aware movement, and correcting accumulated errors over extended exploration.

·2 min
Evaluation & benchmarks

Bayesian framework exposes hidden differences among seemingly similar RAG systems

The authors evaluate 27 retrieval-augmented generation configurations using a Bayesian model structured around how information and errors flow through a RAG pipeline. They show that systems with similar aggregate scores can behave differently after successful versus failed retrieval, and that labeling retrieval success is especially valuable for estimating whether generators follow the intended response policy.

·2 min
Evaluation & benchmarks

Financial agents can cite rules while still attempting prohibited trades

Luo and colleagues built a controlled financial-compliance environment and a monitoring benchmark to test whether language-model agents ground their actions in executable rules. Across experiments with DeepSeek V4 Pro and Gemini 3.5 Flash, displaying rules reduced but did not eliminate rejected orders, while independent monitors could be misled by trader rationales when enforcement evidence was withheld.

·2 min
Evaluation & benchmarks

New benchmark standardizes evaluation of coding agent harnesses

The authors introduce Claw-SWE-Bench, a benchmark and adapter protocol for evaluating OpenClaw-style agent harnesses (claws) on coding tasks. They show that adapter design dramatically affects performance: OpenClaw with a minimal adapter scores only 19.1% Pass@1, while a full adapter achieves 73.4% with the same backbone model. The benchmark also reveals that model choice and harness choice each account for roughly 30 percentage points of variation in performance.

·2 min