Scientists and engineers want (LLMs) to write code that simulates physical systems, like how a bridge bends under load. These problems demand numerically accurate results, not just code that runs without crashing. The difficulty is that even plausible-looking code can be wrong.
The authors extended a to 17 difficult tasks involving nonlinear mechanics and structural dynamics. They tested 15 LLMs both generating code in one shot and using agentic systems that let the model run, inspect, and revise its own code. They also checked whether the models could write tests that catch errors.
The best model succeeded in 87.1% of single-attempt implementations, but when asked to write tests that detect wrong code it reached only 49.9%. Agentic systems improved consistency on many tasks yet still failed to catch their own mistakes; in 73.1% of failed cases the tests simply did not cover the required behavior. The most reliable approach used separate coder and tester agents, achieving 88.8% success across three attempts.
