Training a with is like a factory where machines both produce data (rollouts) and then learn from it. Currently, the factory stops all production until the slowest machine finishes, wasting time and compute power. TIDE is a system that lets machines start learning on completed data while others are still producing, and it moves leftover work to fewer machines to keep them running efficiently. This achieves up to 1.36 times faster overall training and recovers nearly all lost performance, without needing special tuning for each task.
New system recovers wasted GPU time in RL training of language models
TIDE dynamically repurposes idle inference GPUs for training, achieving up to 36% speedup without breaking on-policy guarantees.
Top University
Mathew Jacob · Morgan Borjigin-Wang · Kan Zhu · Po Hao Chen · Yi Pan · Andy Ruan · +7 more
Stanford University · University of California, Berkeley
Research Digest··2 min read

In one line
TIDE recovers GPU utilization lost to straggler rollouts and inefficient decoding in on-policy LLM reinforcement learning training.
Before you read this
Three ideas this paper builds onreinforcement learningA machine learning method where an agent learns to make sequences of decisions by receiving rewards or penalties for its actions.293 papers →large language modelA neural network trained on massive text corpora to generate coherent text and perform diverse language tasks.319 papers →speculative decodingA method that uses a lightweight model to propose token sequences that are then verified in parallel by a larger model.58 papers →
Read it as
The authors introduce TIDE, a system for reinforcement learning post-training of large language models that recovers GPU utilization lost when inference engines stall waiting for straggling rollouts.
The authors identified two sources of GPU under-utilization in synchronous colocated RL training: idle GPUs waiting for straggler rollouts (up to 30% of runtime) and inefficient decoding as batch sizes shrink (up to 9.8% loss).
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.