The incident, which occurred during the ongoing war with Iran, began when a U.S. Special Operations Command analyst queried an AI chatbot to synthesize open-source data with classified signals intelligence. The chatbot inaccurately identified the ship's cargo manifest. The analyst then used the same tool to format the erroneous findings into an official-looking summary, which was circulated across command channels.
One source told CNN the incident "almost started a war." The false intelligence suggested the Chinese vessel was transporting nuclear arms program components through the Middle East. The military was preparing to intercept and board the ship with air support before officials traced the report back to the chatbot and determined it was "entirely false."
The near-miss highlights a growing concern among military officials and outside experts: as decision-makers lean more heavily on AI, errors produced by these systems can travel up the chain of command before being questioned. The Pentagon has described AI as delivering a significant advantage in speeding up its "kill chain," but the same speed may allow hallucinations to pass with insufficient human oversight.
Jake Steckler, research scholar at GovAI and a veteran U.S. Army officer, said in a written response to TechCrunch: "It's important for service members to understand the uncertainty inherent to LLMs. But it's especially critical for any decisions that could lead to use of force, like targeting, intelligence analysis, or operational planning. There are life and death consequences for those decisions."
Steckler added that the incident should serve as a call to add more safeguards, not a reason to avoid AI entirely. "Prioritizing adoption speed over all else will likely lead to incidents that only make service members lose trust in these systems, which ultimately is only going to slow adoption," he said.