AWS says Continuum AI security agent passes 89% of benchmark tasks

Tool finds and patches vulnerabilities in C/C++ code, but testing limited to memory-safety bugs; critic questions timing.

By LineZotpaper AI staff

Published24 October 2026

Read Time2 min

AWS has made its AI-powered security agent Continuum generally available, claiming it achieves an 89% end-to-end success rate at finding and fixing vulnerabilities in open-source projects. The company published benchmark results showing Continuum passed 819 of 920 tasks on the CyberGym-E2E test, a significant improvement over the previous high of 65.9%.

Continuum, released Thursday, scans repositories for code vulnerabilities, validates them against a user's AWS environment, and opens pull requests with proposed fixes. The CyberGym-E2E benchmark gives agents 90 minutes to find and fix a vulnerability in one of 139 open-source C or C++ projects, without a bug description or proof of concept. The agent must identify the flaw, trigger it, and produce a fix that stops the crash without breaking existing tests.

AWS reports Continuum found and reproduced a bug in 92.5% of tasks, stopped the crash in 89.6%, and kept tests passing in 89.0%. However, only 37.8% of patches also fixed the specific historical vulnerability the benchmark targeted, a gap AWS acknowledges: fixing a different real vulnerability does not make the work worthless. The agent is not told which historical bug to fix.

Corey Quinn, chief cloud economist at Duckbill Group, responded to AWS CEO Matt Garman's announcement on X, questioning whether the release was originally planned for AWS re:Invent and was moved up. Quinn offered no evidence for the claim.

AWS ran the tests itself and says Continuum followed the benchmark's rules, including limits on internet access. The company also verified the agent did not look up existing fixes. With more time, AWS says the success rate rises to 93.7%. The benchmark and testing tools are public, but results remain self-reported.

Continuum agent success rates on CyberGym-E2E benchmark, by stage

The final stage requires fixing the specific historical bug the benchmark targets; the agent is not told which bug that is. Source: AWS via The New Stack

Analysis

Why This Matters

Automated vulnerability patching could reduce the burden on security teams, but effectiveness in real-world environments remains unverified.
The benchmark is limited to memory-safety bugs in C and C++, leaving many languages and vulnerability types untested.
Self-reported results without independent verification raise questions about real-world performance.

Background

AWS Continuum is a new service that uses AI to find and fix code vulnerabilities. It targets developers working in cloud environments. The CyberGym-E2E benchmark is a public test designed to evaluate AI agents on end-to-end vulnerability detection and repair. AWS CEO Matt Garman described AI as "genuinely transformative for defenders."

Key Perspectives

AWS: Continuum scored higher than any other AI agent on the CyberGym-E2E benchmark, with an 89% end-to-end success rate. The agent finds and fixes real vulnerabilities, even if it does not always match the historical bug the benchmark checks. CEO Matt Garman said AI is "genuinely transformative for defenders."

Corey Quinn (Duckbill Group): Responded on X questioning whether the release was originally slated for AWS re:Invent and pushed out early given the state of the internet. Quinn offered no evidence.

What to Watch

Independent tests or third-party audits of Continuum's performance on the public benchmark.
How well Continuum performs when directed to fix a specific vulnerability, which the benchmark does not test.
AWS's next announcements at re:Invent, if any, regarding Continuum.

Sources

AWS says its AI security agent fixes 89% of bugs — but there’s a catch · The New Stack

Checked

12 claims in this story checked against 1 source: 12 supported.

Show each claim

SupportedContinuum passed 89% of benchmark tasks, according to AWS.The New Stack: “In that run, Continuum passed 819 of 920 tasks on CyberGym-E2E, an 89.0% end-to-end success rate”
SupportedAWS made Continuum generally available on Thursday.The New Stack: “AWS made Continuum for code vulnerabilities generally available on Thursday.”
SupportedAWS’s result was 819 of 920 tasks, compared with a previous public high of 65.9%.The New Stack: “Continuum passed 819 of 920 tasks on CyberGym-E2E, an 89.0% end-to-end success rate, against a previous public high of 65.9%.”
SupportedCyberGym-E2E gives an agent 90 minutes to find and fix a vulnerability in one of 139 open-source projects, without a bug description or proof of concept.The New Stack: “CyberGym-E2E gives an agent 90 minutes to find and fix a vulnerability in one of 139 open-source projects, without a bug description, proof of concept, crash log, or existing patch to work from.”
SupportedAll benchmark tasks involve memory-safety bugs in C or C++.The New Stack: “All tasks involve memory-safety bugs in C or C++, leaving out many other languages and vulnerability types, as AWS acknowledges.”
SupportedContinuum found and reproduced a bug in 92.5% of tasks.The New Stack: “Continuum found and reproduced a bug in 92.5% of tasks”
SupportedContinuum stopped the crash in 89.6% of tasks.The New Stack: “stopped the crash in 89.6%”
SupportedContinuum kept the project’s tests passing in 89.0% of tasks.The New Stack: “kept the project’s tests passing in 89.0%.”
SupportedOnly 37.8% of tasks had a patch that also fixed the benchmark’s historical vulnerability.The New Stack: “Continuum’s patch also fixed that historical bug in 37.8% of tasks”
SupportedCorey Quinn questioned whether AWS had moved the release up from a planned re:Invent announcement, and offered no evidence for that claim.The New Stack: “Corey Quinn, chief cloud economist and co-founder of Duckbill, replies to the post, wondering whether the release started as a re:Invent announcement and was pushed out early, given the state of the internet. Quinn offered no evidence that AWS had moved up the announcement.”
SupportedAWS ran the tests itself and says Continuum followed the benchmark rules, including limits on internet access; AWS also checked that it did not look up already-public fixes.The New Stack: “AWS ran the tests itself and says Continuum followed the benchmark’s rules, including limits on internet access. The company also checked that the agent wasn’t simply looking up fixes that were already public.”
SupportedAWS says Continuum’s success rate rises to 93.7% with more time.The New Stack: “Give it more time, AWS says, and that rises to 93.7%.”