The paper introduces TraceBench, a simulation-based framework for generating controlled root-cause attribution tasks from physical dynamical systems. Agents receive time-series observations and must determine if a system parameter was altered. Four LLM agents are evaluated on three mechanical systems under varying noise, domain context, and labeled examples.
Key findings show that agents benefit substantially from domain context, prefer numerical console output over visualizations, and perform worse when required to produce a Python script for per-sample labeling versus submitting direct predictions. This provides controlled insights into how LLM agents integrate system descriptions with iterative data analysis for time-series diagnostics.