Problem: Standard combination of separately trained task experts () is expensive, requiring the combined model to generate new examples and experts to score each one.
What was done: The authors propose multi-expert replay policy distillation (MRPD), which reuses the examples experts generated during training, replaying them in the order they were seen without new environment calls.
Main finding: MRPD produces a better combined model while costing about a seventh of the compute used to train the experts. Example order matters more than quantity or rewards, making training logs a curriculum.
