Fri 13 Nov
LiveGet it free
ZOTPAPER

News without the noise

← All concepts

audit workflow

An audit workflow is a systematic process for checking whether a system or dataset is as reliable as it seems. Instead of just running a standard test, you deliberately probe for hidden flaws: mislabeled data, shortcuts that let a model cheat, missing context that changes a result, or rare dangerous behaviors that ordinary evaluations miss. The goal is to uncover what a quick glance would not see.

Across these papers, audits repeatedly show that standard evaluations paint an overly rosy picture. Benchmarks contain label errors and duplicate examples, models rely on spurious clues like page numbers, and evaluations themselves are fragile, small changes in analysis rules can shift rankings drastically. Audits designed to stress-test systems, isolate variables, or simulate adversarial tampering reveal vulnerabilities that simpler checks hide. The results argue for audit workflows with multiple controls, sensitivity analyses, and transparent reporting, because what looks like a reliable score often masks deep problems.

Papers
108
Released code
7
First seen
June 2026
Latest
Nov 2026

99 papers in the last two months, against 6 in the two before.

Start here

Three papers to read first, in the order they came out
  1. 1Benchmark scoring code, not the model, can swing accuracy 22 points

    Scoring rules in multimodal LLM benchmarks can inflate accuracy by up to 22 points on identical outputs.

    Nov 2026
  2. 2Hidden bugs in evaluation code silently reorder AI agent rankings

    Agent leaderboard scores are silently distorted by the grading, logging, cost and versioning code behind them; cheap audit checks can expose the reordering.

    Nov 2026
  3. 3Benchmark errors inflate and deflate AI policy-reasoning scores

    Policy-based decision benchmarks have defects that misjudge rule-following responses, impacting scores and comparisons.

    Nov 2026

How fast it is moving

JunJulAugSeptOctNov so far

Often studied with

verificationChecking outputs for correctness before accepting them, often to catch errors or reward hacking.28 togetheragent safetyRisks from AI agents including timing errors, shutdown avoidance, collusion, overclaiming, and unsafe execution, mitigated by constrained decoding, safety harnesses, and swarm governance.20 togetheragentic evaluationMeasuring AI agents' performance in complex tasks, highlighting failures in planning, monitoring, and real19 together

Who is working on it

Independent Researcher 5Carnegie Mellon University 5Tsinghua University 3Anthropic Fellows Program 3University of Pennsylvania 3National University of Singapore 3

The papers

Most central to this idea first, not most recent.

Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.