Choosing how to align a (LLM) to human preferences currently involves a lot of guesswork. Practitioners try different methods and hope for the best. There are many techniques, each with its own settings and failure modes, so understanding why they work is an open problem.
This paper provides a theory that unifies the most popular methods. It shows each method can be described by a simple equation governing how the model changes which outputs it prefers. This equation has a solution that can be written down exactly, revealing deep differences: some methods cause the model's confidence to grow without limit, while others converge to a fixed level of confidence.
The theory makes three predictions that are confirmed across five types of models on four standard benchmarks. It also leads to a practical improvement: adding a small amount of label smoothing to the SimPO method creates a stable operating point that achieves the highest average instruction-following score among comparison methods on all three tested models (each with 8 to 9 billion parameters).