Language models can lose accurate outcome probabilities when trained on comparisons
Shi and colleagues show that training a language model to agree with truthful comparisons can make its outcome probabilities less accurate. In a controlled robot-pushing simulation, a standard preference-training objective moved 33 to 43 percentage points of probability onto outcomes that never occurred in the enumerated reference distributions.









