CTE-Bench tests whether models can predict how a code edit or state overwrite changes a running service. It provides service code, prior interactions, the intervention, and 40 fixed future calls, then checks predicted responses by executing the modified service.
Across 255 scenarios and six deterministic Python services, four models achieved 54.3% to 61.5% exact match on intervention-affected calls when correct earlier responses were shown. Without that feedback, scores fell to 23.2% to 33.2%, and no model predicted more than 1.2% of scenarios exactly end to end.
The separates prediction from action selection and focuses scoring on calls whose responses actually change. Its central result is that intermediate execution feedback matters much more than the differences among the evaluated models.