AI Chatbot Nearly Triggered Military Action Against Chinese Vessel, Pentagon Report Shows
A U.S. Special Operations Command analyst relied on a chatbot to assess intelligence about a Chinese ship, resulting in a fabricated assessment of nuclear weapons components that almost prompted a military interception.
The United States came close to intercepting a Chinese vessel after receiving what turned out to be entirely fabricated intelligence, according to reporting by CNN. An analyst working for U.S. Special Operations Command submitted the flawed assessment, which claimed the ship was moving components for a nuclear weapons program across the Middle East. The military had begun preparing to board the vessel with air support when officials realized the chatbot used to generate the report had misidentified the cargo.
One person with knowledge of the incident characterized the near-catastrophe to CNN as something that "almost started a war."
How the Error Occurred
The analyst had turned to a chatbot to process intelligence documents detailing what the Chinese ship was carrying. According to CNN, the tool combined publicly available information with classified signals intelligence held by the government, then synthesized this into a formal intelligence assessment. The resulting document nearly triggered a military operation based on material the chatbot had simply fabricated.
This incident represents one of the most serious consequences yet of AI systems generating false information when they lack sufficient training data. Since the term "hallucinating" entered mainstream usage—becoming the Cambridge Dictionary's word of the year in 2023—AI-generated falsehoods have compromised work across numerous fields. Book authors, journalists, researchers, judges, physicians, law enforcement agencies, and corporate customer service operations have all been misled by language models that invent details rather than acknowledge gaps in their knowledge. Some researchers have suggested that preventing these errors may be fundamentally impossible with current large language model technology, despite various attempts to instruct systems to avoid making things up.
The Pentagon's embrace of generative AI tools has accelerated even as these risks have become clearer. In January, the Department of Defense unveiled an "AI acceleration strategy" designed to "make all appropriate data available across federated IT systems for AI exploitation, including mission systems across every service and component." Defense Secretary Pete Hegseth stated at the time, "AI is only as good as the data that it receives, and we're going to make sure that it's there."
Military AI Deployment
The Pentagon has been rapidly integrating commercial AI systems into its operations. In December, the Department of Defense announced it would build its "GenAI.mil" platform on Google's Gemini for Government. The following month, the department added Grok for Government as an additional option. Anthropic provides a customized version of Claude designed for U.S. intelligence applications.
A Pentagon official told Congress in June that the military uses generative AI to help produce reports required by Congress, and that 1.5 million active Department of Defense personnel have accessed military generative AI tools.
Existing Safeguards
In 2023, the State Department released a "Declaration on Responsible Military Use of Artificial Intelligence and Autonomy" that called for armed forces to use AI in a "principled" manner. The declaration emphasized the importance of "careful consideration of risks and benefits" and the need to "minimize unintended bias and accidents." It also stressed that responsible AI deployment must include "a human in the loop, a responsible human chain of command and control."
Despite these stated principles, the military has continued moving toward greater autonomy in weapons systems. Fully autonomous attack drones have been deployed in the conflict in Ukraine and tested by military contractors working with NATO. In March, the Department of Defense removed Anthropic from its approved vendor list, citing the company's refusal to allow its models to be used in autonomous weapons. A federal judge ruled last month that this action constituted "unlawful retaliation in violation of the First Amendment."
The near-miss with the Chinese ship comes amid growing alarm from AI researchers about existential risks posed by advanced systems, sparking renewed debate over AI safety regulation and calls for coordinated research oversight among leading AI development companies.