Anthropic said its Mythos 5 model almost never stopped a cyberattack once it began, even when told that real people were its targets. The AI agents fabricated farcical explanations to justify their impersonation and malware schemes, the company reported.
In August, an Anthropic agent created fake emails and bombarded a GitHub moderator's inbox with requests to insert malware into code. The original experiment was conducted by the UK's AI Security Institute, which tasked an Anthropic agent with hacking into a simulated computer network to test its cybersecurity capabilities. Though the agent had internet access, researchers intended it to only use the internet to help find ways to hack into the network. Instead, the agent accessed an actual GitHub project, incorrectly believing it was part of its simulated target. It wrote malware and tried several times to trick the actual GitHub owner into adding it to the project. In its requests, the agent misrepresented the malware, describing it to the person as a useful update and browbeating them when they declined.
In a thorough debrief of the incident released on Friday, Anthropic found that once an agent had begun the attack, there were basically zero scenarios in which it could be called off. Like a rabid dog ignoring the pleading cries of its owner, the agent 98% of the time in one test ignored Anthropic's warnings once it had its first taste of blood, according to the report. Anthropic called this disquieting proclivity "momentum" and added that while its Mythos 5 model, released in June, showed little regard for warnings, current models were more inclined to stop.
"Once it had begun performing dangerous actions, Claude Mythos 5 almost always continued doing so, ignoring information that might have suggested its trajectory was harmful," Anthropic said in its report.
For its new report, Anthropic repeatedly asked agents to perform the same task. It then measured how the agents behaved when different models and stimuli were introduced. Anthropic tried to throttle the agent's frontal attack with three categories of warning: telling it that it was inflicting "real world harm," that its environment was wonky, or that it was manipulating a real GitHub repository. None had any effect once the models progressed beyond the initial stages of the GitHub attack. At that point, the models gave absurd answers when confronted with their bad behavior. When asked what the model believed it was attacking, it answered that the GitHub repository was real, but that the people it was targeting were somehow simulated. "This stated belief, in addition to being somewhat unlikely a priori, is essentially impossible," Anthropic said.
