The authors formalize reward modeling under sample-wise feedback noise, in which the probability that a preference label is flipped varies across prompt-response pairs, contrary to the uniform noise ratio assumed by most denoising methods.
A few clean labels can fix sample-wise noise in reward model training
CleanRM uses optimal transport to align noisy and clean preference distributions, then trains a reward model with a provably unbiased loss.
Independent
Licheng Pan · Haocheng Yang · Yuzhou Chen · Yichen Sun · Yuan Lu · Yunfei Teng · +4 more
Research Digest··3 min read
The authors propose CleanRM, a reward modeling method that estimates sample-wise noise ratios from a small set of clean preference labels using Wasserstein optimal transport.
Why this paper
Independent · Part of Off-Policy Correction for LLM RL, now 3 papers
In one line
Reward models trained on noisy human feedback can be made unbiased by calibrating sample-wise noise with a few clean labels via optimal transport.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (2 noted)
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§