Reinforcement learning for LLM agents often suffers from sparse outcome-only rewards that do not credit individual steps. T2SPO addresses this by reusing successful trajectories from the agent's own training history. For each visited state in a new rollout, a frozen TabPFN regressor predicts the remaining distance to success, using past state-representation and completion-length pairs as context. The change in predicted distance between consecutive states provides auxiliary step-level credit alongside the final outcome reward.
Experiments with 1.5B and 7B language models on the ALFWorld and WebShop benchmarks show that T2SPO consistently improves overall task success compared to the GRPO baseline. The method is notable because it does not require training a separate value model—the regressor remains frozen while the context is updated with newly completed trajectories. This makes step-level practical with minimal overhead.