Visual generative models, like image generators, are often improved by a fine-tuning step called (RL) to make their single outputs better. But in practice, these models are typically used by generating many candidates and picking the best one, a method called Best-of-N sampling.
This paper studies whether the RL improvement carries over when using Best-of-N sampling. The authors found that on hard tasks, RL can actually make things worse: as they sampled more images, the original model eventually outperformed the fine-tuned one.
The core discovery is that RL tends to improve the average image while hurting the rare, very best outcomes, exactly the ones Best-of-N relies on. This means evaluating models on single samples is misleading; we need to check how RL reshapes the entire distribution of possible outputs.