The authors categorize process reward uncertainty into two sources: PRM uncertainty, from limited supervision data and imperfect training, and policy uncertainty, from mismatch between the strong policy that generated Monte Carlo labels and the weaker policy used during test-time scaling.
Bayesian belief framework jointly handles two uncertainties in process reward models
Reliability-aware updating and conservatism-aware calibration improve reward accuracy and test-time scaling on multimodal reasoning benchmarks.
Chinese Tech
Zhongjian Qiao · Zhixiang Zhou · Jiafei Lyu · Peisong Wang · Qiaosheng Zhang · Siyang Gao · +1 more
CityUHK · Shanghai Innovation Institute · Tencent · Institute of Automation, Chinese Academy of Sciences
Research Digest··2 min read
The authors propose BayesianPRM, which treats process rewards as random variables and maintains a Bayesian belief over a discretized reward hypothesis space.
§