Video typically assigns a single reward to an entire video, but different parts of the video may have different quality. TVRL improves this by deriving token-level credit from the same reward model: a frozen (VLM) that answers questions about the video. The VLM's gradients indicate which video tokens most affect the reward, allowing the policy update to focus on problematic tokens.
The method is applied to (GRPO) for video . It averages rewards from multiple question-answer checks into a shared advantage and uses gradient magnitudes as weights for log-probabilities inside the clipped policy ratio. This preserves GRPO structure while adding spatial and temporal credit allocation.
On VBench-2.0, TVRL outperforms vanilla GRPO by 1.33 to 3.15 points across different reward models and samplers, showing consistent improvement in aligning generated videos with prompts.