Fri 13 Nov
LiveGet it free
ZOTPAPER

News without the noise

← All concepts

benchmark

A benchmark is a standardized testing setup that evaluates how well a system performs on specific tasks, using consistent inputs and scoring criteria. It allows for fair comparisons across different models or approaches and often reveals strengths and weaknesses that simpler tests might overlook.

The introduced benchmarks cover areas like conversational privacy, agent forecasting, and identity consistency in images. Their findings reveal that current models often fail in nuanced ways, such as leaking secrets after topic shifts, losing identity across edits, or performing differently across interfaces, which standard evaluations miss. This underscores the need for benchmarks that reflect realistic conditions and for models built with explicit mechanisms to handle these challenges.

Papers
347
Released code
13
First seen
June 2026
Latest
Nov 2026

335 papers in the last two months, against 10 in the two before.

Start here

Three papers to read first, in the order they came out
  1. 1AI agents fall for social engineering attacks in every tested scenario

    All frontier AI models fail to resist social engineering attacks, with proactive verification below 32% and many making ungrounded action claims.

    Nov 2026
  2. 2Coding agents control simulated robots less reliably under realistic constraints

    The strongest coding agent succeeds only 34% of the time on robot tasks under real-world constraints.

    Nov 2026
  3. 3Generalist tabular models match specialists on utility, not fidelity

    Foundation models show strong performance across generation, imputation, and prediction on tabular data.

    Nov 2026

How fast it is moving

JunJulAugSeptOctNov so far

Often studied with

agentic evaluationMeasuring AI agents' performance in complex tasks, highlighting failures in planning, monitoring, and real138 togetherverificationChecking outputs for correctness before accepting them, often to catch errors or reward hacking.77 togethervision-language modelA model that handles tasks combining visual and textual information, such as editing images from conversation or reasoning across images and text.67 together

Who is working on it

Zhejiang University 14Alibaba Group 13Ant Group 12NVIDIA 11Carnegie Mellon University 10Amazon 10

The papers

Most central to this idea first, not most recent.

Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.