The authors formulate Generalized Proximal Policy Optimization (GPPO) using f-divergences, a broad family of measures for differences between probability distributions.
Divergence geometry and estimation jointly improve language-model policy optimization
GPPO places several policy-optimization algorithms in one framework and finds that curved divergences paired with lower-variance estimators produce stronger reasoning models.
Chinese Tech
Zhicheng Cai · Xinyuan Guo · Hanlin Wu · Weinan Dai · Yuyan Zhou · Wei-Ying Ma · +2 more
Institute for AI Industry Research (AIR), Tsinghua University · SIA-Lab of Tsinghua AIR and ByteDance Seed
Research Digest··3 min read
Cai and colleagues recast proximal policy optimization for language models as a choice along two axes: how policy change is measured and how that change is estimated from samples.
Why this paper
From SIA-Lab of Tsinghua AIR and ByteDance Seed and Institute for AI Industry Research (AIR), Tsinghua University
In one line
GPPO unifies LLM RL algorithms via divergence and estimator choices, outperforming standard GRPO by up to 50% on math and 14% on coding.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks (2 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§