The authors constructed 450 tasks from 150 real-world vulnerability scenarios.
Coding agents produce functional code that often fails security tests
Across 450 tasks derived from real vulnerabilities, six frontier agents rarely generated code that was both correct and secure.
Chinese Tech
Tianyi Wu · Chao Ge · Peng Chen · Fengji Zhang · Yongxing Wu · Yuchong Xie · +9 more
National University of Singapore · Qwen Team, Alibaba Token Hub, Alibaba Group · Cloud Safety and Security, Alibaba Cloud, Alibaba Group
Research Digest··2 min read
Wu and colleagues introduce LSCBench, a benchmark that tests whether coding agents address security risks when users specify only desired functionality.
Why this paper
From Qwen Team, Alibaba Token Hub, Alibaba Group and 2 others
In one line
Frontier coding agents usually produce functional code but pass both functional and security tests on at most 25.1% of realistic vulnerability tasks.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§