Evaluation Metric Validity
Investigates the statistical reliability and sensitivity of default metrics used to evaluate generative models across modalities.
10 papers
Where this stands
The written synthesis of this thread is for subscribers. Subscribe.
How this thread developed
Oct 2026 · Warsaw University of Technology
Default kernel metrics for generative models miss rotation changes at common sample sizes
Reveals that default kernel-based metrics fail to detect rotation changes, motivating more robust evaluation designs.
2 further papers
Oct 2026
AI model rankings can reverse when prompts include solved examples
Shows that benchmark rankings reverse when prompts include solved examples, questioning metric robustness and fairness.
1 further paper
Oct 2026 · Technical University of Munich
Batching makes AI grading verdicts depend on the other items
Shows LLM-judge pass/fail verdicts flip 11-22% based on which items share a batch, exposing sensitivity of grading metrics to evaluation setup.
1 further paper
Nov 2026 · Universität Zürich, ETH Zürich
AI hiring tools favor some writing models over others, not candidate quality
Reveals a stable evaluator-model preference hierarchy in AI-mediated hiring, showing LLM judges score writers by model identity rather than candidate quality.
1 further paper
Nov 2026 · Allen Institute for AI, New York University
Language models struggle with state tracking beyond the simplest cases
Introduces a four-level hierarchy for state tracking and tests current models, highlighting evaluation metric validity for a specific reasoning capability.
5 of 10 papers shown