The authors built tasks from two sources: repository histories containing issues that maintainers explicitly linked, and existing single-turn benchmarks converted into staged conversations.
Coding agent rankings change when debugging becomes a longer conversation
Wu and colleagues evaluate eleven agents on 72 repository-level debugging tasks, finding different leaders as the number of conversational turns increases.
Chinese Tech
Zhan-Yu Wu · Ji Luo · Alyssa Wu · Junjielong Xu · Richeng Xuan · Zhichao Hu · +2 more
Tencent Hunyuan
Research Digest··2 min read
Wu and colleagues introduce UserSWE, a benchmark that tests coding agents through conversations with simulated developers whose communication styles change between turns.
§