RL with live server state boosts multi-step tool use in LLMs

PROVE framework combines stateful MCP environments, grounded data synthesis, and programmatic rewards to improve multi-turn tool orchestration across four models.

PaperBig Techcs.CLarXiv:2606.03892v2 ↗

Ibrahim Abdelaziz · Asim Munawar · Kinjal Basu · Maxwell Crouse · Chulaka Gunasekara · Suneet Katrekar · +1 more

IBM Research

Research Digest·3 June 2026·2 min read

Read the paper →

Thread:Credit Assignment in Agentic RL

The authors present PROVE, a framework for reinforcement learning (RL) on multi-step tool use that overcomes three key obstacles: lack of realistic execution environments, ungrounded synthetic queries, and reward-induced verbosity. Training four different LLMs with PROVE yields significant gains on three multi-turn tool-use benchmarks, with improvements up to +10.2 points on BFCL Multi-Turn.

What they did

The authors built a library of 20 stateful MCP servers that expose 343 tools, enabling RL training with live execution and session-scoped state isolation. They designed a state-machine data synthesis pipeline that generates multi-turn tool-call trajectories anchored to the real server state, ensuring that tool calls reference actual entities. They also introduced a multi-component programmatic reward that includes an adaptive efficiency penalty to counteract the verbosity bias of recall-based rewards. Using GRPO, they trained four models (Qwen3-4B, Qwen3-8B, Qwen2.5-7B, Granite-4.1-8B) on approximately 13,000 synthesized examples.

Key findings

On BFCL Multi-Turn, PROVE-trained models improved by up to +10.2 points compared to base models. On tau2-bench and T-Eval, improvements reached +6.8 and +6.5 points respectively.
The adaptive efficiency penalty reduced unnecessary tool calls without sacrificing task completion, as measured by recall and precision metrics.
Training with live-execution rewards outperformed offline reward modeling, due to accurate grounding in the actual server state.

Why it matters

This work addresses a practical bottleneck in training LLMs for realistic tool use: the gap between synthetic training data and real-world server state. By providing a reproducible framework with open-source MCP servers and grounded data synthesis, it enables more effective RL training for multi-step tool orchestration, a key capability for agentic AI systems.

Caveats

The experiments were limited to four relatively small models (up to 8B parameters) and a narrow set of servers (20 MCP servers). The scalability to larger models and diverse tool ecosystems remains untested. Additionally, the state-machine synthesis may not capture all real-world user interaction patterns, and the long-term reliability of learned policies in dynamic live environments is not evaluated.

Analysis

The PROVE framework addresses a known limitation in prior work on tool-use RL, which often relied on static or simulated environments. By grounding training in live server state and introducing an efficiency penalty, the authors mitigate two common failure modes: tool calls that fail because entities don't exist, and excessive tool calls driven by recall-biased rewards. The consistent gains across multiple model families suggest that the approach generalizes, at least within the studied scale. However, the paper does not explore whether these benefits persist with larger frontier models or more complex tool systems, leaving an open question about scalability.

This work also aligns with current trends toward using the Model Context Protocol (MCP) as an interface for tool integration, and the release of the server library could facilitate further research. The reliance on programmatic rewards, while efficient, may not capture nuanced task success criteria that require human judgment, but it is a practical step for automated training.