What they did

The authors analyze three choices in reinforcement learning with verifiable rewards: allocating rollouts according to problem difficulty, organizing samples as a branching tree rather than independent parallel attempts, and selecting branch points using sentence-level entropy, a measure of model uncertainty.

They combine these elements in Difficulty-Adaptive Sentence-entropy-guided Tree-structured Policy Optimization (DATPO). The method also adds a sibling-diversity advantage term that rewards semantically distinct reasoning paths, then evaluates the resulting policy on mathematical reasoning benchmarks.

Key findings

• Difficulty-adaptive rollout allocation expanded pass@k rather than merely reducing computation on easier problems.
• Tree-structured rollouts found correct answers more effectively than parallel sampling in the authors’ comparisons.
• Sentence-entropy-guided branching produced greater semantic diversity than token-level branching, which tended to concentrate branches around localized variations.
• DATPO outperformed the evaluated baselines particularly on pass@k, yielding stronger performance when additional solutions were sampled at test time.

Why it matters

RLVR often raises single-attempt accuracy without broadening the set of reasoning paths a model can successfully pursue. The authors’ results suggest that rollout topology and branch selection are consequential parts of the learning algorithm: explicitly training for diverse successful trajectories can make inference-time sampling more productive.

Caveats

The supplied abstract does not identify the models, benchmarks, compute budgets, baselines or numerical effect sizes, so the magnitude and generality of the gains cannot be assessed here. The evidence is limited to mathematical reasoning, and it remains unclear whether sentence-level tree search offers similar benefits in domains without straightforward verifiable rewards.