This paper unifies three existing regression-based diffusion methods (DiffusionNFT, FlowAWR, and RAM) by showing they each solve a divergence-constrained reward-maximization problem, differing only in the convex generator that defines the constraint. Prior methods made approximations in constructing the advantage-embedded regression target, such as approximating the KKT condition or posterior normalizer, which the authors show can lead to instability or suboptimality. The authors propose DiffusionRFT, which uses the exact sparsemax projection onto the probability simplex for the linear tilt (Pearson χ² generator). Empirically, DiffusionRFT converges faster, trains more stably, and attains top performance compared to prior heuristics. The work clarifies the design space and provides a principled framework for future regression-style diffusion RL algorithms.
Three diffusion RL methods unified as divergence-constrained reward maximizers
Authors show DiffusionNFT, FlowAWR, and RAM share a common framework; their new method DiffusionRFT achieves faster convergence and stability.
Chinese Tech
Toyota Li · David Zhao · Alan Zhao
Tencent
Research Digest··3 min read
In one line
DiffusionRFT uses exact sparsemax projection to converge faster, train more stably, and achieve top performance in regression-based diffusion RL.
Before you read this
Three ideas this paper builds onreinforcement learningA machine learning method where an agent learns to make sequences of decisions by receiving rewards or penalties for its actions.301 papers →diffusion modelA type of generative model that learns to reverse a gradual noising process to create data.199 papers →flow matchingA generative training method that learns an ordinary differential equation mapping noise to data by matching velocity fields along probability paths.175 papers →
Read it as
A unified theoretical framework reveals that three prominent regression-based diffusion reinforcement learning methods—DiffusionNFT, FlowAWR, and RAM—are each solutions to a divergence-constrained reward-maximization problem, differing only in the convex generator defining the constraint.
The authors formalized the optimization objective underlying regression-based diffusion RL: maximize reward on a frozen rollout buffer under a divergence penalty.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.