The authors train a reward head as a linear probe on hidden states from an intermediate layer of a frozen vision-language backbone, with a Bradley-Terry loss over in-batch comparisons.
Reward models for vision-language AI can be trained without human preference labels
A linear probe on a frozen vision-language backbone, trained on same-image perturbations, matches or beats GPT-4o on several reward benchmarks using only a few thousand parameters.
Top University
Yueying Li · Jingxuan Fan · Kianté Brantley · Sham M. Kakade · Hanlin Zhang
Massachusetts Institute of Technology · Cornell University · Scale AI · Harvard University
Research Digest··3 min read
Li et al.
Why this paper
From Harvard University and 3 others
In one line
At 32B, a linear probe on a frozen VLM trained on caption perturbations with no human labels beats GPT-4o on VL-RewardBench.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ✓Compute or model size stated (params 32B)
- ✓Limitations stated by the authors
- ✓Reports numbers on named benchmarks (4 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§