The authors constructed SWE-Shift by saving an agent’s execution state at an intermediate point and then continuing in two parallel ways: a Clean continuation proceeds without any change, while a Shifted continuation receives a task-relevant update (e.g., code edit, dependency change, runtime state alteration, or revised goal).
Coding agents fail when their workspace changes mid-task
New benchmark SWE-Shift shows success on static tasks does not imply robustness to updates during execution.
Independent
Victor Shea-Jay Huang · Rongqing Li · Zhangyue Yin · Kelong Mao · Yaqi Zhao · Xinyuan Wang · +4 more
Research Digest··2 min read
The authors introduce SWE-Shift, a benchmark with 120 scenarios where a coding agent's workspace changes mid-execution, using matched Clean and Shifted continuations from the same saved state.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.