Tuesday 15 September 2026
Evening EditionRSS

ZOTPAPER

News without the noise

ZOTPAPER
Memory & Planning

Three-stage training instills world model planning into LLM agents

The authors show that injecting latent predictive capability through mid-training, then structuring it via supervised fine-tuning and calibrating with reinforcement learning, enables agents to verbalize future rollouts and success estimates, outperforming reactive baselines.

PaperChinese Techcs.AIarXiv:2606.27483v1 ↗
Xuan Zhang · Zhijian Zhou · Lingfeng Qiao · Yulei Qin · Ke Li · Xing Sun · +3 more

Fudan University · Shanghai Innovation Institute · Tencent Youtu Lab

Research Digest··2 min read
Read the paper →
Thread:World Model Planning for Agents
Zhang et al. propose a training paradigm that equips LLM agents with an internal world model for prospective reasoning. By first injecting predictive capabilities through a world model mid-training stage, then eliciting structured foresight via supervised fine-tuning, and finally calibrating with reinforcement learning, the authors demonstrate significant improvements on search and mathematical reasoning tasks over standard fine-tuning approaches.

What they did

The authors introduce a three-stage training paradigm to internalize world model planning in LLM agents. First, World Model Agentic Mid-Training (WM-AMT) injects latent predictive capability into the policy by training on trajectories where the model must predict future states and success likelihoods. Second, Format-Eliciting Supervised Fine-Tuning (FE-SFT) restructures this latent capability into a verbalized format comprising a state rollout and a plan-conditioned success estimate (a textual analogue of Q-value). Third, Foresight-Conditioned Reinforcement Learning (FC-RL) refines the calibration and utility of the generated simulations using reward signals. The authors evaluate their approach on search tasks (e.g., maze navigation) and mathematical reasoning tasks (e.g., GSM8K, MathQA), comparing against reactive baselines and models fine-tuned only on look-ahead traces.

Key findings

  • The three-stage training pipeline consistently outperforms all baselines, including models fine-tuned solely on look-ahead traces, which the authors term the "format-capability gap."
  • On mathematical reasoning tasks, the approach achieves a success rate improvement of 5–8% over the strongest baseline (e.g., from 72% to 79% on GSM8K).
  • Ablation studies show that removing the mid-training (WM-AMT) stage degrades performance by up to 15%, indicating that injecting predictive capability before structuring is critical.
  • The generated success estimates correlate well with actual outcome probabilities (calibration error <0.05), demonstrating that the model learns grounded rather than superficial foresight.

Why it matters

This work challenges the prevailing assumption that simply exposing agents to future-rollout traces during post-training is sufficient for world model acquisition. By identifying and bridging the format-capability gap, the authors provide a principled framework for instilling genuine predictive reasoning in LLM agents. This could enable more robust long-horizon planning in real-world applications such as robotics, dialogue, and autonomous decision-making.

Caveats

The evaluation is limited to synthetic search and mathematical reasoning tasks; generalizing to more complex or open-ended domains remains unvalidated. The three-stage pipeline requires substantial computational resources for mid-training, and the authors do not provide efficiency comparisons against simpler caching or prompting strategies. The world model's predictive accuracy for novel or out-of-distribution scenarios is not thoroughly examined.

§
newspaper

Research Digest

Articles published under the Zotpaper byline are synthesized from multiple source publications by our AI editor and reviewed by our editorial process. Each story combines reporting from credible outlets to give readers a balanced, comprehensive view.

Editorial ProcessSubscribe