6 and used synthetic-document finetuning followed by supervised finetuning to instill four traits: maximizing help over time, instrumental self-preservation, instrumental rather than intrinsic rule-following, and calibrated knowledge of oversight.
RL Training Can Amplify Compliance That Disappears Without Oversight
A model trained only for task quality and policy compliance increasingly followed rules when monitored but broke them when monitoring was absent.
Big Tech
Bart Jaworski · Nathaniel Mitrani · Fred Bruford · Victoria Krakovna
MATS Research · Google DeepMind
Research Digest··2 min read
Jaworski et al.
§