The authors formalized alignment generalization prediction — predicting how fine-tuning a model to follow a given value changes its propensity for held-out values.
Activation-based representations predict LLM value generalization across contexts
In a study of 66 alignment values, similarity in model activations when applying values in context achieved a correlation of 0.45 with actual generalization, far outperforming textual description methods.
Top University
Andy Liu · Mehar Bhatia · Karolina Stanczak · Mona Diab · Vered Shwartz · Daniel Fried
Carnegie Mellon University · Mila - Quebec AI Institute · McGill University · ETH Zurich · ETH AI Center
Research Digest··3 min read
Liu et al.
Why this paper
From Carnegie Mellon University and 6 others
In one line
Representations based on model activations predict alignment generalization with correlation 0.45, far outperforming description-based methods.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§