A language world model predicts what might happen after an action, like a robot push. It needs to know how often rare bad outcomes occur. Asking a judge which of two outcomes seems more realistic is a practical way to train it. This is called comparative feedback.
This study simulated a robot push with known true outcome probabilities. They started with an accurate language world model and trained it further using Bradley-Terry feedback from a truthful judge. They compared the model's final probabilities to the true ones.
Training with Bradley-Terry shifted 33 to 43 percentage points of the model's probability onto outcomes that never actually happen. This distorted the hazard probability. The shift comes from the training objective, not the comparisons themselves, because a different method using the same comparisons avoided the shift.
