Wu and Wei constructed six safety datasets, covering truth judgement, sycophancy, policy violation, general harm, jailbreak and prompt injection, each built from minimal pairs and containing up to 10^6 labels.
Safety probe accuracy follows predictable scaling laws based on model size and label count
Experiments with six new datasets and 20 language models show probe error falls as a power law in model size and label count, allowing planning before a probe is trained.
Big Tech
Yuchen Wu (University of Washington) · Jerry Wei (Anthropic)
Research Digest··3 min read
The authors train safety probes on 20 language models from five families using up to 10^6 labels per task, and find that probe error, defined as 1 - AUROC, follows two power-law scaling laws in model size and label count.
§