From July 21 to August 11, one employee from each of METR and Redwood Research was given access to OpenAI's internal codebase, documentation and some technical staff, the evaluators said in a release. The employees were chaperoned on site and used company-issued devices that never left the premises. The pilot was designed for OpenAI to test what it would look like to give external researchers a deep level of access to its internal infrastructure. The evaluators discovered several novel vulnerabilities, including an exploit that would allow agents to prevent hundreds of recent actions from being monitored. OpenAI told the evaluators it has patched those vulnerabilities that impact its production safeguards, according to the release. METR and Redwood said they agreed to publish only a high-level public post because the pilot was not designed to provide robust public transparency about OpenAI's monitoring system. They shared a 61-page internal report with OpenAI and said they expect to share more details publicly about future exercises. An OpenAI spokesperson said in a statement to CNBC: "This collaboration builds on our longstanding commitment to pioneering new approaches to third-party testing, and we're using what we learned to strengthen our safeguards and shape future collaborations with independent evaluators." The pilot comes after OpenAI CEO Sam Altman pledged in September to grant independent evaluators employee-like access, following a proposal by Anthropic CEO Dario Amodei as part of his plan to slow the pace of AI development. Anthropic has since announced agreements with organizations including Accenture, METR and SecureBio. OpenAI said earlier this month that it is actively finalizing contracts with third-party safety assessors. President Donald Trump, who has taken a light-touch approach to AI regulation, has also supported third-party assessment, encouraging AI companies to partner with an independent external auditor or evaluator as part of a voluntary accord he presented at the White House. Anthropic's Amodei signed the accord, as did top executives at Google, Meta, OpenAI, SpaceX and Nvidia. Independent evaluators have historically occupied a relatively quiet corner of the AI industry, and experts told CNBC that questions remain about how these groups will be funded and what their reporting structures will look like. METR and Redwood said they stress-tested OpenAI's internal misalignment monitoring system during the pilot. In AI, alignment refers to whether a model behaves in line with human interests and values. The evaluators assessed the monitoring system as it existed on and before July 21 and noted that OpenAI was expanding the system during the exercise.
Third-party evaluators find vulnerabilities in OpenAI monitoring system during pilot
Redwood Research and METR cite exploit that could prevent monitoring of hundreds of recent actions
How it's landing
This exercise was designed to pilot giving external researchers deep infrastructure access, not to provide the kind of public transparency our recent proposal envisions. For example, we agreed upfront to only publish a brief, high-level output like we’ve done in similar pilots.
Posts on X from people the desk follows. Their words, unedited.
Analysis
Why This Matters
- The finding of vulnerabilities in OpenAI's monitoring system raises questions about the robustness of internal safeguards at leading AI companies, even before external access is fully granted.
- The pilot marks a step toward third-party evaluation becoming standard practice in AI safety, but the limited public disclosure and lack of broad transparency highlight ongoing tensions between corporate confidentiality and independent oversight.
- As OpenAI and others finalize contracts with safety assessors, the structure and funding of these evaluations will determine whether they can effectively catch dangerous capabilities before deployment.
Background
Third-party AI safety evaluation has emerged as a key element of voluntary and government-backed frameworks for managing frontier model risks. Following proposals from Anthropic CEO Dario Amodei, OpenAI CEO Sam Altman pledged in September 2026 to give independent evaluators employee-like access to internal systems. The White House under President Donald Trump has encouraged such partnerships but has not mandated them. METR and Redwood Research are among a small group of nonprofits that conduct technical safety testing of AI systems, but their work has historically operated on limited budgets and without deep access to proprietary code.
Key Perspectives
OpenAI: The company presented the pilot as a learning exercise and said it has patched discovered vulnerabilities. It is actively contracting with third-party assessors and sees the collaboration as building toward stronger safeguards. METR and Redwood Research: The evaluators stressed that the pilot was not designed for public transparency, but they identified real vulnerabilities and shared a full internal report with OpenAI. They expect future exercises to produce more public findings. Critics: Independent evaluators have limited funding and unclear reporting structures, raising concerns about whether they can provide sustained, rigorous oversight without being co-opted by the companies they assess.
What to Watch
- Whether OpenAI and other signatories of the White House accord make third-party evaluation reports public or keep them internal, setting a precedent for industry transparency.
- The details of contracts being finalised between OpenAI and safety assessors, especially funding levels and reporting obligations.
- Any future exploitation of the type of vulnerability found in the pilot, which could indicate whether fixes are comprehensive.
Sources
Checked
10 claims in this story checked against 1 source: 9 supported.
Show each claim
- SupportedThird-party evaluators found vulnerabilities in OpenAI’s monitoring system during a pilot.Tech: “Nonprofit evaluators METR and Redwood Research on Thursday said they conducted a pilot review of an internal monitoring system at OpenAI”
- SupportedMETR and Redwood Research cited an exploit that could prevent hundreds of recent actions from being monitored.Tech: “The evaluators discovered several “novel vulnerabilities,” including an exploit that would allow agents to prevent “hundreds” of recent actions from being monitored.”
- Partly supportedOpenAI said it had patched the issues found in the pilot.Tech: “OpenAI told METR and Redwood that it has patched the identified vulnerabilities that impact its production safeguards, according to the release.”
- SupportedThe pilot ran from July 21 to August 11, with one employee from each evaluator given access to OpenAI’s codebase, documentation and some technical staff.Tech: “From July 21 to Aug. 11th, an employee from each organization had access to OpenAI's internal codebase, documentation and some technical staffers.”
- SupportedThe employees were chaperoned on site and used OpenAI-issued devices that never left the premises.Tech: “The employees were chaperoned while on site and used OpenAI-issued devices that “never left the premises,” according to a release.”
- SupportedThe evaluators shared a 61-page internal report with OpenAI.Tech: “They shared a 61-page internal report with OpenAI, and said they expect to publicly share more details about future exercises.”
- SupportedAn OpenAI spokesperson made the quoted statement about using the collaboration to strengthen safeguards and shape future collaborations.Tech: ““This collaboration builds on our longstanding commitment to pioneering new approaches to third-party testing, and we're using what we learned to strengthen our safeguards and shape future collaborations with independent evaluators,” an OpenAI spokesperson told CNBC in a statement.”
- SupportedSam Altman pledged in September to give independent evaluators employee-like access after Dario Amodei proposed the idea as part of a plan to slow AI development.Tech: “OpenAI CEO Sam Altman pledged to grant independent evaluators “ employee-like access ” to the company in September, after Anthropic CEO Dario Amodei proposed the idea as part of his plan to slow the pace of AI development .”
- SupportedTrump encouraged AI companies to partner with an independent external auditor or evaluator as part of a voluntary accord he presented at the White House.Tech: “He encouraged AI companies to “partner with an independent external auditor or evaluator” as part of a voluntary accord he presented at the White House.”
- SupportedMETR and Redwood said they stress-tested OpenAI’s internal misalignment monitoring system, assessing it as it existed on and before July 21 while OpenAI was expanding it.Tech: “METR and Redwood said Thursday that they stress-tested OpenAI's internal misalignment monitoring system during its three-week-long pilot. ... The evaluators assessed the monitoring system as it existed on and before July 21, and noted that OpenAI was expanding the system during the exercise.”