The authors constructed tasks by subtraction: from a paper's released code, they deleted the key contribution, keeping the deleted code as the human reference.
Benchmark tests LLM agents on recreating published computer science contributions
FrontierCS 2 places agents inside a working repository and judges their patches against the original authors' code on held-out workloads.
Independent
Qiuyang Mang · Wenhao Chai · Runyuan He · Dongruixuan Li · Zihan Zheng · Yujun He · +55 more
Research Digest··3 min read
The authors introduce FrontierCS 2, a benchmark of 48 open-ended tasks across 13 computer science domains.
§