When a server crashes while processing a refund, the code must leave data in a consistent state. Many applications rely on database transactions to ensure that either all steps complete or none do. LLMs are often evaluated by running code once in a single environment, which can mask whether the code itself is safe or is depending on a library default.
The researchers built a test that crashes code at each database write, then re-runs it as an operator would. They did this in two modes: Python's default sqlite3 mode (which silently opens a transaction) and autocommit mode (where every write saves immediately). They tested 9,100 solutions from ten models on 24 tasks.
More than half (54%) of database code that survived crashes in the default mode left data half-updated in autocommit mode. This held for every model. Asking for crash safety helped, but most code only became safe when explicitly instructed, revealing that much apparent safety is borrowed from the library.