SWE-Flux is a new benchmark for evaluating whether LLMs can reason about how programs actually execute, not just what code means statically. It contains 480 execution-grounded instances drawn from 12 real Python repositories, with gold answers harvested automatically from instrumented test executions rather than written by humans or judged by LLMs. The benchmark covers single-test and multi-test questions spanning control flow, loops, program state, dataflow, exceptions, and program invariants.
Five LLMs were evaluated, and the best reached only 37% accuracy. Models did relatively better on localized behaviors such as invariants, intra-procedural control flow, exceptions, and simple loops, but struggled with dataflow, inter-procedural execution, precise state reasoning, and suite-level aggregation. The benchmark pipeline can also generate fresh variants through input perturbation, succeeding for almost 90% of selected instances, and these variants were substantially harder for the models. This matters because coding agents increasingly need runtime reasoning, and existing benchmarks mostly test static understanding or isolated snippets.