(RLVR) improves math reasoning in language models, but its effectiveness depends on the diversity of the rollout trajectories. Simply adding search to generate more diverse trajectories does not guarantee a better policy. The authors propose APIVIS, a training-time framework that adapts Gumbel search to select better reasoning chunks during each rollout. It mixes direct and searched responses in each training group and uses selective supervision on tokens improved by search. This ensures that even when all responses in a group have the same final reward (all correct or all wrong), the model still receives a useful learning signal from the search-improved tokens. Experiments across multiple math benchmarks with Qwen3-1.7B and Qwen3-4B models show consistent improvements over both direct RLVR and search-based methods.
Value-guided search improves math reasoning during RL training
APIVIS adapts Gumbel search to chunk-level reasoning, mixing direct and searched responses to create learnable rewards.
Big Tech
Shaohuai Liu · Yuning Wu · Haoran Liu · Enzo Jia · Devin Chen · Kai Wei
Amazon · Texas A&M University
Research Digest··2 min read
In one line
APIVIS uses value-guided search during training to improve math reasoning beyond what independent sampling achieves.
Before you read this
Three ideas this paper builds onreinforcement learning with verifiable rewardsReinforcement learning that uses rewards from objectively checkable signals like correct answers or verification feedback.163 papers →group relative policy optimizationA reinforcement learning method that optimizes policy by computing advantages based on group-level reward statistics.188 papers →verificationChecking outputs for correctness before accepting them, often to catch errors or reward hacking.341 papers →
Read it as
The authors propose APIVIS, a training-time framework that integrates finite-budget Gumbel search into reinforcement learning with verifiable rewards (RLVR) for mathematical reasoning.
The authors addressed the limitation that rollout diversity alone does not guarantee policy improvement in RLVR.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.