Merging specialist AI models saves 42% of training compute compared to joint training

Training reasoning models often uses reinforcement learning (RL), which is compute-heavy. On-policy distillation (OPD) is a different method that might be cheaper. This study compares OPD with Group Relative Policy Optimization (GRPO).

The authors measured accuracy versus total floating-point operations (FLOPs) for both methods, varying teachers and domains. They tested external teachers and using the student's own GRPO checkpoint as teacher.

OPD with external teachers needed more FLOPs to reach GRPO's accuracy. But with the student's own GRPO checkpoint as teacher, OPD matched that accuracy using 50 times fewer FLOPs. Multi-teacher OPD merged specialists into one model, saving 42% total FLOPs over joint GRPO training. This makes OPD a cheap consolidation tool.