38 papers this week in Evaluation & benchmarks12 active threadsbusiest: Alignment Training Dynamics

Evaluation & benchmarks research

Evaluation & benchmarks

AI agents can comply with visual task specs but lag on design quality

The authors introduce VisualArtifactSpec 1.0, a benchmark of 211 creation and modification tasks for AI agents covering presentations, static graphics, animations, and web interfaces. Evaluating nine model-harness configurations, they find overall scores range from 42 to 70 out of 100, with Artifact Performance (assessing content, organization, visuals, and delivery) accounting for 73.4% of the gap between the highest and lowest scores. High scores on Task Fulfillment, checking explicit requirements, frequently coexisted with poor Artifact Performance, showing that meeting specifications does not guarantee effective design for the intended audience.

today
Evaluation & benchmarks

AI models now recite test answers from memory, inflating scores

Benchmark contamination has escalated from indirect optimization to explicit memorization. Qin et al. document that recent base models can reproduce entire test examples, questions, options, and reference answers, that were never in their input, with one model recalling 96% of a coding benchmark. They propose an Anti-Hack score that subtracts credit for leaked examples and show it moves a leading model to last place.

today