Benchmark shows coding agents struggle to synthesize static-analysis checkers
The authors introduce CheckerBench, an executable benchmark of 300 real-world tasks for long-horizon static-analysis checker synthesis, and CheckerLab, a unified evaluation framework. Across 21 model-harness configurations, the mean Pass@1 is 32.30%, with the best configuration reaching 45.33%. The results indicate that reliable, reusable checker development remains difficult for current coding agents.
