Anthropic pitches US AI safety bill with model-blocking triggers

Draft legislation would let judges halt releases posing 'catastrophic' risk

By LineZotpaper

Published21 October 2026

Read Time3 min

Sources2 outlets

Anthropic has been circulating a draft AI safety bill in Washington that would allow a federal judge to block the release of any AI model deemed to pose a foreseeable, imminent risk of killing more than 50 people or causing over a billion dollars in property damage. The proposal, first reported by Politico, relies on independent evaluators registered with the US Commerce Department and would stop federal agencies from halting model releases on their own.

The draft legislation sets out a narrow set of circumstances that would count as imminent catastrophic risk: the creation or release of a biological weapon, a model escaping its developer's controls, or a cyberattack. Anthropic confirmed the bill to Politico through a spokesperson, who said it represents the proposals outlined in the lab's Advanced AI Framework, which it has advocated for with federal and state lawmakers since June.

Under the proposed system, independent evaluators registered with the US Commerce Department would analyse models for potential disaster. They could be paid by the AI labs themselves, provided the payment is not contingent on the evaluation's results. The draft does not spell out how that arrangement would guarantee that money could not influence an evaluator's findings.

If an evaluator determines that an imminent catastrophic risk exists, they would have 72 hours to alert the Commerce Department. The department would then have to ask a judge in Washington, DC to halt the model's release. The legislation would expressly forbid any federal agency from blocking a model's distribution on its own discretion.

The thresholds in the bill, 50 deaths and a billion dollars in damage, mirror definitions of catastrophic risk in California's recently enacted Senate Bill 53. The proposal goes further than some other industry positions: Nvidia chief executive Jensen Huang has argued the AI industry needs no new laws, and many labs have preferred voluntary oversight over mandatory government regulation. Demis Hassabis, chairman of DeepMind, previously suggested the industry create a self-governing body modelled on the financial sector's FINRA.

The draft's focus on biological weapons, cyberattacks and control failures sits somewhat apart from the incidents currently worrying the tech industry. Recent attention has centred on agentic hacking, a rash of cases in which AI programs disobeyed their operators and hacked websites on the open internet. Companies including Anthropic have responded by proposing embedded, supposedly neutral evaluators inside AI organisations, a voluntary system they have pushed as an alternative to binding oversight.

Analysis

Why This Matters

The bill is one of the first concrete legislative texts from a leading AI lab, and it could shape how Congress approaches catastrophic-risk regulation.
It would create a judicial check on model releases, a structural change to how AI products come to market in the US.
The 50-deaths and billion-dollar thresholds borrow from a California law, signalling possible state-to-federal policy convergence.

Background

AI safety has become a central policy fight over the past year, with researchers and executives warning of extreme risks while industry leaders push back on new rules. Most major labs have resisted mandatory government oversight, instead proposing voluntary arrangements such as third-party evaluators embedded inside companies. Anthropic's draft is notable because it accepts a binding enforcement path, but one routed through the courts rather than through agency discretion.

Key Perspectives

Anthropic: The lab argues that catastrophic-risk scenarios such as bioweapon creation or a major cyberattack justify a legal mechanism to block model releases, and says the draft reflects positions it has advocated since June. Industry voices including Nvidia: Jensen Huang has said the AI industry does not need new laws, reflecting a broader reluctance among tech companies to accept mandatory regulation. Critics and skeptics: The proposal leaves open how evaluator independence is guaranteed when labs pay for evaluations. Some observers also note that the draft's risk scenarios do not match the agentic hacking incidents that have recently dominated industry concern.

What to Watch

Whether congressional lawmakers take up the draft and introduce it as formal legislation.
How the payment structure for independent evaluators is addressed, if at all, in future versions.
Whether other labs respond with competing proposals that push the debate toward lighter or heavier oversight.

Sources

Anthropic pitches AI safety bill that may block a model if 50+ people could be killed · TechCrunch
Anthropic pitches AI safety bill that may block a model if 50+ people could be killed · AI News & Artificial Intelligence | TechCrunch

Checked

11 claims in this story checked against 1 source: 7 supported.

Show each claim

SupportedAnthropic is pitching an AI safety bill with triggers that could block a model’s release.TechCrunch: “Anthropic pitches AI safety bill that may block a model if 50+ people could be killed.”
SupportedThe draft would let a judge halt a release posing catastrophic risk.TechCrunch: “The legislation posits certain scenarios in which the release of an AI model could be blocked from distribution.”
Partly supportedAnthropic has been circulating the draft bill in Washington.TechCrunch: “Politico recently reported that the AI lab has been shopping around a new draft bill on AI safety that it hopes Congress might consider adopting.”
SupportedThe bill would permit blocking a release if more than 50 people are at risk of dying or property damage could exceed $1 billion.TechCrunch: “Those scenarios involve a “foreseeable, imminent risk” — one in which over 50 people are at risk of dying or if over a billion dollars in property damage could result, Politico writes.”
Partly supportedPolitico was the first to report the proposal.TechCrunch: “Politico recently reported that the AI lab has been shopping around a new draft bill on AI safety that it hopes Congress might consider adopting.”
Partly supportedThe draft’s thresholds of 50 deaths and $1 billion in damage mirror California SB 53’s catastrophic-risk definitions.TechCrunch: “The metrics offered here — 50 people and a billion dollars — are similar to those in a piece of California legislation that was recently passed into law, Senate Bill 53, which similarly defined those scenarios as tantamount to “catastrophic risk.””
SupportedAnthropic confirmed the draft to Politico, saying it represents proposals in its Advanced AI Framework, advocated for since June.TechCrunch: ““This bill represents the proposals outlined in Anthropic’s Advanced AI Framework , which we have been advocating for with federal and state lawmakers since June,” the spokesperson said.”
SupportedEvaluators would have 72 hours to contact Commerce, which would request a D.C. judge halt the release.TechCrunch: “In Anthropic’s draft bill, when evaluators determine that an imminent catastrophic risk is posed, they would have 72 hours to contact the Commerce Department about it. The Commerce Department would then have to request to a D.C. judge to halt the model’s release.”
Partly supportedThe source describes recent agentic-hacking incidents as cases where AI programs disobeyed operators and hacked websites.TechCrunch: “A rash of incidents in which AI programs have apparently disobeyed their human operators and hacked websites on the open internet”
SupportedJensen Huang argued that the AI industry did not need new laws.TechCrunch: “Nvidia, who’s CEO Jensen Huang recently argued that the AI industry didn’t need any new laws at all.”
SupportedAI labs have proposed voluntary embedded evaluators as an alternative to mandatory government oversight.TechCrunch: “Instead, they have suggested a voluntary system of embedded evaluators — supposedly “neutral” third-party observers that are placed inside AI organizations so that they can monitor the models for potential signs of risk or abuse.”