Psychological harm from AI conversations is hard to detect directly because it builds over time and depends on the user. Instead of measuring harm, researchers looked for specific AI behaviors that can lead to harm, such as claiming emotions or fostering attachment.
They defined eight such behaviors and built a synthetic dataset of 47,062 AI responses labeled for them. They compared simple classifiers, direct LLM judging, and a two-stage cascade where a cheap classifier screens then an LLM judge reviews positives. Human annotations validated the approach.
The cascade achieved a 0.79 F1 score (a measure combining precision and recall) and cut LLM judge calls by 97.1% on real conversations. This allows continuous monitoring of AI behavior without accessing model internals, offering a scalable way to spot psychological risk early.