In recent weeks, leaders of major AI companies have publicly raised alarms about the technology they are building. Anthropic CEO Dario Amodei argued that AI carries serious risk and that progress should be slowed. OpenAI CEO Sam Altman responded, saying, "I agree with Dario that we need to pace the frontier."
The warnings came after AI researcher Jacob Coxon announced he was leaving Anthropic, charging that neither it nor OpenAI was acting responsibly. "The people building AI earnestly believe that it could kill us all by the end of the decade," he posted on X. Another Anthropic employee, Evan Hubinger, agreed publicly: "We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade."
One specific fear is that AI could help design and release a bioweapon—a highly lethal virus targeting people by their genes, a crop-killing fungus, or an undetectable toxin. In 2022, researchers at Collaborations Pharmaceuticals demonstrated the ease of misuse using an AI "molecule generator" originally developed to find potential drugs. In less than six hours, the model generated 40,000 molecules with potential as chemical warfare agents, some more toxic than known nerve agents. The authors called it "a wake-up call for our colleagues in the 'AI in drug discovery' community."
David Magnus, a professor of medicine and biomedical ethics at Stanford University, said the study was very scary. "Of course, everything since then has just sort of blown up."
Today, large language models can answer questions spanning all scientific fields, trained on the knowledge of "almost every scientist who ever lived," says Dunja Sabra, a biosecurity researcher at the University of Hamburg. These models can provide instructions and video training for experiments. Combined with accessible gene-editing and synthetic biology tools—enabling DIY biology labs—the situation could become very dangerous. "The chances are that someone determined would succeed eventually," Sabra says.
The article noted that safeguards exist for building new genomes, but did not detail them. The full MIT Technology Review piece explores further implications.