for flow matching models can degrade output quality because reward-driven updates push the sample trajectories off the pretrained data manifold. The authors identify this failure mode as manifold drift and show theoretically that a preference update leaves the manifold whenever the terminal displacement has a nonzero normal component. To counter this, they propose ThermoDPO, a temperature-controlled objective that anchors the optimization on preferred samples, and a weighted variant for low-temperature settings. On a toy benchmark ThermoDPO-weighted achieves a StrictScore of 0.899, compared to 0.629 for FlowDPO, and on the SD3.5-M image model it improves OCR by 47.5% and the average of four metrics by 16.0%.
Temperature-controlled preference updates reduce manifold drift in flow models
The authors identify off-manifold movement as a source of reward hacking and propose an objective that better preserves pretrained sample structure.
Top University
Yansen Han · Shengyi Liao · Yuanxing Zhang · Pengfei Wan · Tao Lin
Zhejiang University · Kuaishou Technology · Westlake University
Research Digest··2 min read
In one line
Manifold drift in flow preference optimization moves terminal samples off the pretrained manifold, causing reward hacking.
Before you read this
Two ideas this paper builds onRead it as
Han et al. analyze preference optimization for flow-matching generative models, showing that reward-driven updates can push outputs away from the pretrained data manifold when the induced displacement has a component normal to that manifold.
The authors formalize “manifold drift,” in which preference updates alter a flow model’s transport trajectories so that terminal samples leave the support of the pretrained data distribution.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.