SynthDemo-RL is a teacher-student framework that replaces human demonstrations with synthetic ones for fine-tuning Vision-Language-Action (VLA) models. An automated teacher, using simulator-privileged state and LLM-guided adjustments, generates successful manipulation trajectories. These are distilled into a VLA student via supervised fine-tuning (SFT), which is then refined with PPO using binary task-success rewards.
On LIBERO-PRO, a benchmark of perturbed tasks with no existing demonstrations, 27 of 57 tasks had 0% success for a baseline policy. Direct PPO rescued only 10, leaving 17 at 0%. SynthDemo-RL, using 50 synthesized trajectories per task and no new human demonstrations, rescued all 27 and achieved average success rates of 97.8% and 97.1% on the Position and Task axes. On standard LIBERO, it reached 96.0% with no human demonstrations, within 1.7 points of a baseline trained on 50 human demonstrations per task.
This work is interesting because it removes the need for human demonstrations when adapting VLA models to new tasks, overcoming the zero-reward barrier in sparse-reward RL. It shows that synthetic demonstrations can effectively initialize RL at VLA scale, even when the teacher's trajectories are suboptimal.