QF3 is an online off-policy algorithm designed to train flow policies. It combines flow matching with the critic's action gradient, backpropagated through a one-step prediction of the flow's output. To maintain reliable updates, the gradient is applied only to action dimensions that stay close to the replay action from the buffer.
The method is the first off-policy flow RL approach capable of training humanoid locomotion policies from scratch and transferring them zero-shot to real hardware. It achieves a 10x wall-clock speedup over FPO++, a recent on-policy flow RL method. QF3 also fine-tunes pretrained flow-based manipulation policies on ABC-Sim and Robomimic tasks, showing that it can both learn policies from scratch and refine those learned from demonstrations.