The authors evaluated assumptions behind alignment midtraining, or AMT: continued pretraining on large collections of alignment-relevant text before later finetuning.
Alignment midtraining does not reliably generalize desired model behavior
Tests at up to 110 billion parameters found that midtraining effects were fragile and could not substitute for explicit behavioral demonstrations.
Industry
Sid Baines · Jonathan Bostock · Maria Angelica Martinez · Andrew Draganov · David Africa · Daniel Tan
Arcadia Impact · Resolution
Research Digest··2 min read
Thread:Safety Training Side Effects
Baines et al.
Why this paper
From Arcadia Impact and Resolution
In one line
Midtraining steers model motivations in simple ambiguous settings, but a few percent of conflicting finetuning data erases its effects.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§