What they did

The authors built TraceBench, a framework that simulates physical dynamical systems and generates controlled root-cause attribution tasks. For each task, an agent receives time-series observations and must decide whether a system parameter changed during the simulation and, if so, identify that parameter.

They generated tasks from three interpretable mechanical systems and evaluated four LLM agents under varied experimental conditions. They also recorded agent trajectories to examine how the agents explored and analyzed the data.

Key findings

• Providing domain context substantially improved agents’ root-cause attribution performance.
• Agents examined time-series data mainly through numerical console output rather than visualizations.
• Agents generally performed worse when asked to write a Python program assigning a root-cause label to every sample than when allowed to submit predictions directly.
• The benchmark covers both detection—whether a parameter changed—and attribution—which parameter was responsible.

Why it matters

TraceBench provides a controlled way to study agent behavior on time-series diagnosis, where real-world datasets often lack reliable ground truth about causal changes. Its simulations make it possible to isolate how domain knowledge, analysis strategy, and output format affect performance rather than relying only on aggregate accuracy from observational data.

Caveats

The evaluation is limited to three simulated, interpretable mechanical systems and four agents, so its conclusions may not transfer directly to noisy industrial, medical, or infrastructure data. Simulation also simplifies issues such as interacting faults, incomplete observations, sensor failures, and ambiguous real-world causes.