Temperature-controlled preference updates reduce manifold drift in flow models

The authors identify off-manifold movement as a source of reward hacking and propose an objective that better preserves pretrained sample structure.

PaperTop Universitycs.AIarXiv:2608.20011v1 ↗

Yansen Han · Shengyi Liao · Yuanxing Zhang · Pengfei Wan · Tao Lin

Zhejiang University · Kuaishou Technology · Westlake University

Research Digest·23 August 2026·2 min read

Read the paper →

What they did

The authors formalize “manifold drift,” in which preference updates alter a flow model’s transport trajectories so that terminal samples leave the support of the pretrained data distribution. They prove that optimal flow matching recovers the terminal data distribution, while a preference update departs from the pretrained manifold whenever its terminal displacement has a nonzero normal component.

They then introduce ThermoDPO, a temperature-controlled pairwise preference objective anchored on preferred samples. Varying the temperature connects the method to rejection-sampling fine-tuning and FlowDPO, while controlling a pointwise, reconstruction-based surrogate for distance from the manifold. A weighted version, ThermoDPO-weighted, is designed to restore weak optimization signals at low temperatures.

Key findings

On the main toy benchmark, ThermoDPO-weighted achieved a StrictScore of 0.899, versus 0.629 for FlowDPO and 0.857 for FlowDPO combined with rejection-sampling fine-tuning.
On Stable Diffusion 3.5 Medium at classifier-free guidance strength 4.5, the weighted method improved optical character recognition performance by 47.5%.
Across four reported SD3.5-M metrics, it improved the average score by 16.0%.
The theoretical analysis ties off-manifold movement specifically to the normal component of the preference update’s terminal displacement, rather than to reward optimization in general.

Why it matters

The work offers a geometric explanation for one form of reward hacking in continuous-time generative models: optimizing a preference score can exploit directions unsupported by the pretrained distribution. Constraining or penalizing those directions could make preference tuning more reliable without discarding the pretrained model’s learned structure.

Caveats

The strongest quantitative result comes from a toy benchmark, while the larger-model evidence is limited to SD3.5-M at one reported guidance setting. ThermoDPO controls a reconstruction-based surrogate rather than manifold distance directly, and the abstract does not establish how well that surrogate generalizes across architectures, reward models, datasets, or more extensive human evaluation.