Anthropic alignment lead backs departing researcher's warning: AI existential risk is 'correct'

In a social media thread Tuesday night, Jacob Coxon announced his departure from Anthropic, warning that frontier AI companies are 'gambling with our lives' by developing systems that they 'earnestly believe... could kill us all by the end of the decade.' Coxon attributed the existential risk not to current models but to the impending prospect of 'self-improving superintelligence' creating 'superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.'

He expressed concern that others working on these models have either not 'internalized the civilizational stakes' or believe they need to 'speedrun' the race to superintelligence to prevent an irresponsible party from getting there first.

Lending significant weight to Coxon's warning, Hubinger responded on social media to say that 'Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.'

The episode stands out in a pattern where high-profile AI researchers often depart frontier labs to pursue startups or protest business models. Coxon's public warning, bolstered by a senior colleague's agreement, has intensified ongoing debates about AI safety and the pace of development at leading labs.

Why This Matters

A senior safety researcher at a leading frontier AI lab has publicly quantified an existential risk from AI at over 10% within a decade, lending institutional credibility to warnings often dismissed as science fiction.
The departure and public statement signal potential internal divisions at Anthropic about the safety of fast-paced AI development, even as the company positions itself as a safety-conscious alternative to competitors.
The debate now shifts from abstract risk to a concrete timeline and probability estimate coming from inside the industry.

Background

Anthropic has long positioned itself as a safety-focused AI lab, founded by former OpenAI employees who left over concerns about OpenAI's direction. The company's core mission involves building AI systems that are 'helpful, honest, and harmless.' Coxon's resignation and Hubinger's concurrence highlight tensions within even the most safety-conscious labs: the pressure to compete with rivals such as OpenAI and Google may be forcing development of capabilities that researchers themselves view as potentially catastrophic.

Key Perspectives

Jacob Coxon (departing researcher): Argues that the race to self-improving superintelligence poses an existential threat and that the AI industry is speeding toward a dangerous outcome without adequately internalizing risks.
Evan Hubinger (Anthropic Alignment Science lead): Publicly agrees with Coxon's assessment, stating a personal belief that AI extinction risk exceeds 10% within 10 years, lending internal credibility to the warning.
Critics/Skeptics: Some AI researchers and industry observers argue that existential risk claims are overblown or that safety measures can keep pace with capability advances. Others note that public estimates of this nature could fuel regulatory backlash without improving safety practices.

What to Watch

Whether other Anthropic researchers or executives publicly comment on the warning, particularly CEO Dario Amodei or co-founder Jack Clark.
Any changes in Anthropic's public safety protocols or development pace following the internal dissent.
Policy reactions from regulators or lawmakers who may seize on the probability estimate to justify new AI legislation.

ZOTPAPER