The authors derived a state-distribution correction from the off-policy policy gradient, which allows learning from trajectories generated by an earlier behavior policy.
Trajectory-aware updates stabilize reinforcement learning for longer LLM reasoning
TAPO corrects policy mismatch accumulated across generated trajectories, improving reported math and coding performance while limiting training collapse.
Chinese Tech
Zhicheng Cai · Weinan Dai · Yuyan Zhou · Wei-Ying Ma · Ya-Qin Zhang · Hao Zhou
Institute for AI Industry Research (AIR), Tsinghua University · SIA-Lab of Tsinghua AIR and ByteDance Seed
Research Digest··3 min read
Cai and colleagues identify a weakness in long-horizon LLM reinforcement learning: small policy changes at each token can compound, causing the updated model to encounter states poorly represented by the policy that generated its training data.
Why this paper
From SIA-Lab of Tsinghua AIR and ByteDance Seed and Institute for AI Industry Research (AIR), Tsinghua University
In one line
Correcting trajectory-induced state distribution shift in LLM RL improves long-horizon reasoning performance and training stability.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks (2 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§