The authors assembled 619 repository-level tasks spanning ten programming languages from SWE-bench Pro, SWE-bench Multilingual, and DeepSWE.
Generated Software Tests Often Fail to Distinguish Correct Patches
Across 619 repair tasks, evaluating tests against thousands of candidate patches exposed weaknesses hidden by checking only the reference fix.
Chinese Tech
Hua Lu · Pengcheng Wang · Han Li · Zichen Tian · Yanyu Chen · Huang He · +1 more
Baidu Inc. · Nanjing University · The Chinese University of Hong Kong
Research Digest··3 min read
Lu and colleagues introduce SWEPatchTest, a benchmark that evaluates whether agent-generated tests reproduce an issue, accept correct solutions, and reject incorrect ones.
Why this paper
From Baidu Inc. and 2 others
In one line
Generated tests that reproduce a reference fix often fail to distinguish correct from incorrect patches; validation-guided training roughly triples joint validation performance.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (2 noted)
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§