The authors address the efficiency bottleneck in RL post-training for large language models, where rollout generation accounts for around 80% of total training time and exhibits high variability due to diverse response lengths and multi-turn interactions.
Experience replay boosts RL training efficiency for large language models by reusing trajectories
ReFlow, a decoupled and adaptive system, achieves up to 2.06× throughput improvement over existing methods while maintaining model quality.
Chinese Tech
Yukun Tian · Junwei Su · Haofeng Xu · Lansong Diao · Zhengping Qian · Weibo Zhao · +3 more
The University of Hong Kong · University of Science and Technology of China · Alibaba Group · Southeast University
Research Digest··3 min read
The authors propose ReFlow, a system-algorithm co-designed reinforcement learning framework that uses experience replay of recent trajectories to amortize the high cost of rollout generation in LLM post-training.
Why this paper
From Alibaba Group and 3 others
In one line
ReFlow reuses recent trajectories and dynamically reallocates resources to accelerate LLM reinforcement learning by up to 2.06 times without reducing model performance.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§