The authors studied preference optimization, which trains models to score preferred responses above rejected alternatives.
Shared hidden-state drift can collapse language generation while preference rankings improve
Li and colleagues trace repeated-token collapse to a shared representation shift and reproduce the failure by injecting that shift into an untreated model.
Chinese Tech
Yun Li · Ehsan Javanmardi · Simon Thompson · Jiachen Gong · Qunli Zhang · Zifan Zeng · +4 more
The University of Tokyo · TIER IV, Inc. · Huawei
Research Digest··2 min read
Li and colleagues show how preference training can improve response-ranking metrics while pushing a language model toward uncontrollable repetition.
§