On-policy reinforcement learning (RL) often becomes unstable when target-domain data is scarce, yielding noisy gradient estimates. This paper introduces MFPG-PPO, a multi-fidelity policy gradient method that leverages abundant, cheap low-fidelity data (e.g., from a simplified simulator) as a control variate to reduce variance in policy gradient estimates computed from limited high-fidelity data (e.g., real robot rollouts). Unlike methods that directly optimize biased objectives from low-fidelity data, MFPG-PPO remains unbiased with respect to the high-fidelity objective.
Experiments across simulated robot locomotion tasks and a physical Franka arm show that MFPG-PPO improves upon standard PPO trained on high-fidelity data alone in nearly all settings. On the hardest task with the smallest high-fidelity budgets, it matches the performance of PPO trained with 16x more high-fidelity data. Most importantly, MFPG-PPO enables stable learning on a real robot using only 4 episodes per policy update, with no human demonstrations. The method is robust to low-fidelity data bias and scales monotonically with more high-fidelity data.