SecProbe is a framework that uses (IRT) to adaptively select or synthesize vulnerability-repair tasks for evaluating . It estimates agent ability and identifies where additional evidence is most informative, reducing the number of tasks needed while maintaining discriminative power. The system can generate new tasks on demand when existing ones are insufficient.
In experiments, the authors constructed 353 tasks spanning six programming languages and 151 Common Weakness Enumeration (CWE) types, and evaluated nine frontier models with two . The highest success rate observed was 28.33%, indicating substantial gaps in vulnerability recognition and repair. Compared to random and one-shot baselines, SecProbe achieved comparable ability estimates while requiring agents to solve up to 29.5% fewer tasks.
This work is interesting because it addresses the problem of static benchmarks becoming outdated as agents evolve, and it offers an efficient, adaptive approach that concentrates evaluation effort on unresolved capability gaps. The results highlight that current coding agents are far from reliable at handling cybersecurity vulnerabilities.