The study analyzed 450,000 gender-directed completions from 15 GPT models across generations. It found that explicit harmful content like sexual violence disappeared by GPT-4, but new forms of bias emerged, such as framing breast cancer as a men's rights issue in men-directed output. Standard toxicity classifiers scored this new content as non-toxic, showing that harm reduction metrics can miss representational harm.
Safety training can launder gender discrimination into subtler forms, study finds
Analysis of 450,000 GPT-model completions shows explicit toxicity falling while representational harm grows, evading standard classifiers.
Academic
Sarah Wyer · Sue Black · Noura Al Moubayed
Durham University
Research Digest··3 min read
Thread:Safety Training Side Effects
In one line
Gender discrimination in GPT models is transformed into subtler forms rather than removed by safety training.
Before you read this
An idea this paper builds onRead it as
Researchers at Durham University analysed 450,000 gender-directed text completions from 15 GPT models (GPT‑2 through GPT‑5) and found that safety training reduces overtly toxic language but transforms discrimination into subtler, non‑toxic biases.
The authors collected 450,000 sentence completions from 15 OpenAI GPT models (GPT‑2 to GPT‑5) using prompts that directed each model to complete sentences about women or about men (three demographic conditions).
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.