The authors evaluated a single frozen System One model (cost ~USD 0.000025 per call, latency ~100 ms) on 142,658 comments from the Civil Comments corpus, which includes toxicity as the fraction of human raters who flagged each comment.
Cheap moderation model matches frontier LLM accuracy but conflates disagreement and decision risk
First independent audit of a System One model on 142,658 comments shows it is accuracy-competitive with 13 of 14 frontier models at far lower cost, but its confidence fails to capture human annotator disagreement.
Big Tech
Karan Yadav
Research Digest··2 min read
The authors audit a commercial System One content moderation model on the Civil Comments corpus, comparing it to 14 frontier reasoning models.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.