DRM (Diffusion ) replaces the conventional scalar reward head with a diffusion-based head that estimates the conditional distribution of reward given a prompt and response. This allows it to capture the multimodal nature of human preferences without imposing a parametric output family. The model uses a frozen LLM encoder and a lightweight to denoise noise into a reward vector, handling both multi-attribute regression and pairwise preference data.
At inference, multiple samples from the diffusion process form an empirical reward distribution, which can be aggregated into a scalar, variance, or quantiles. Across five benchmarks, DRM matches or surpasses baselines under matched data and backbone, and it stays competitive with much larger discriminative, distributional, and generative RMs despite modest training scale. It also recovers multimodal reward structure where conventional heads collapse to a point.
Uncertainty-aware rejection and lower-confidence-bound aggregation demonstrate that DRM can exploit distributional information to improve reward-model decisions. Downstream RLHF experiments show that using DRM as the training-time reward leads to improved policy performance, validating the practical benefit of diffusion-based reward modeling.