Dense credit assignment from gold-answer log-probabilities improves long-horizon agent RL

TRACE assigns per-turn rewards via temporal-difference changes in log-ratio values, boosting performance on complex search tasks without explicit critics or process supervision.

PaperBig Techcs.LGarXiv:2607.13988v1 ↗

Leitian Tao · Baolin Peng · Wenlin Yao · Tao Ge · Hao Cheng · Mike Hang Wang · +2 more

University of Wisconsin–Madison · Microsoft Research

Research Digest·16 July 2026·2 min read

Read the paper →

Thread:Credit Assignment in Agentic RL

what_they_did

TRACE treats agent rollouts as state transitions at tool-call boundaries. It obtains log-probabilities of the gold answer from a frozen reference model given the current state, transforms them into log-ratio state values, and computes per-action rewards as the temporal-difference (TD) change in those values. This requires no additional critic, process-label supervision, or reward model training. The one-step log-ratio TD component telescopes across redundant tool calls, allowing dense reward signals even in long trajectories. The method is evaluated on the closed-web BrowseComp-Plus benchmark and transferred to open-web tasks, using Qwen3-4B and Qwen3-30B-A3B models.

key_findings

TRACE improves Qwen3-4B from 7.2 to 35.6 and Qwen3-30B-A3B from 8.4 to 42.6 on BrowseComp-Plus, representing multi-fold gains.
It enables pure RL training without a cold-start supervised fine-tuning stage, agentic mid-training, or training on live-web data.
Learned search behavior transfers to open-web benchmarks, and training curves show earlier improvement and faster convergence compared to outcome-only RL.
The method requires no explicit process-reward model or critic network, simplifying the training pipeline.

why_it_matters

Long-horizon agent tasks suffer from sparse and high-variance outcome rewards, making credit assignment a bottleneck for RL-based post-training. TRACE provides a way to assign credit to individual tool calls using only the gold answer's log-probabilities from a frozen reference model, eliminating the need for costly process supervision or learned critics. This could make RL training for agents more sample-efficient, practical, and accessible, especially in domains where outcome signals are available but intermediate supervision is not.

caveats

The method assumes a frozen reference model can compute log-probabilities of the gold answer conditioned on partial trajectories; this may not be feasible for all tasks or model architectures. The experiments focus on web browsing and search tasks (BrowseComp-Plus and open-web benchmarks), so generalization to other tool-use domains (e.g., coding, robotics) is not established. Additionally, TRACE requires the gold answer to be available at the end of each trajectory, which may not hold in all real-world agent deployments.

TRACE sits in a line of work addressing credit assignment for RL agents, offering an alternative to process reward models (PRMs) and outcome-based RL. By leveraging a frozen reference model's log-probabilities – a form of implicit value function – it avoids the need for training a separate critic or labeling intermediate steps. This shares ideas with contrastive methods and success story assignment but is tailored for tool-calling agents. The large gains on BrowseComp-Plus highlight the sensitivity of long-horizon agents to reward sparsity; TRACE's dense signals may unlock RL as a viable post-training method for such tasks, potentially reducing reliance on supervised fine-tuning and large-scale demonstration data.

ZOTPAPER