The authors propose a method to make diagnostic reasoning tasks amenable to reinforcement learning by engineering verifiable rewards. They create a simulator that injects hidden interventions into a controlled digital advertising environment, generating observations and oracle labels. An agent must investigate noisy data using Python and SQL to identify the root cause and affected segment. They train Qwen3.5-35B-A3B with supervised fine-tuning followed by reinforcement learning using these synthesized rewards. On a held-out test set of 235 episodes, the trained model achieves 0.757 FullAttr@1, surpassing the strongest prompted baseline (Claude Opus 5 at 0.686) and a larger prompted model (Qwen3.5-122B at 0.283), while also using fewer tool calls. This result suggests that access to a scalable, objective training signal can be more important than model scale alone for diagnostic reasoning tasks.
Synthetic rewards train agents to diagnose simulated advertising anomalies
TRACE turns hidden simulator interventions into objective rewards for training agents to investigate noisy data with Python and SQL.
Independent
Rui Sun · Zhan Shi · Bing He
Independent Researchers
Research Digest··2 min read
In one line
Reinforcement learning with synthesized rewards from a simulator enables a small language model to outperform larger frontier models at diagnostic reasoning.
Before you read this
Three ideas this paper builds onagentic reinforcement learningTraining approach that lets LLM-based agents learn multi-step tool use and planning via reinforcement learning, often with dense or per-tool rewards.102 papers →failure attributionIdentifying which component or step in a system caused an observed failure, often using structured trace representations.115 papers →agent-driven simulationAutonomous agents automatically generate, control, or extend simulations to improve accuracy, interactivity, or performance.32 papers →
Read it as
The authors built a digital-advertising simulator in which injected interventions provide known labels for otherwise ambiguous diagnostic problems.
The authors created TRACE, a controlled diagnostic environment covering 12 root causes and fine-grained affected-segment attribution.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.