The authors first empirically characterize the Posterior Concentration Phenomenon by analyzing posterior rewards from LLM generations across varying trace lengths.
Posterior rewards become indistinguishable for long reasoning traces in LLM training
The authors identify a length-dependent failure mode they call Posterior Concentration Phenomenon, and propose a framework to mitigate it.
Top University
Shiu-Hong Kao · Yubo Zhao · Zhenyu Tian · Pengzhan Sun · Yicong Li · Angela Yao
National University of Singapore · The Hong Kong University of Science and Technology · University of Science and Technology of China
Research Digest··3 min read
Kao et al.
Why this paper
From National University of Singapore and 2 others
In one line
Probability-based reinforcement learning rewards for LLMs become indistinguishable for long reasoning traces, causing unstable training.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§