An intelligence report circulated within the U.S. military in the spring of this year amid tensions over the war with Iran triggered an immediate state of alert; the report stated that a Chinese ship in the Middle East was carrying components related to a nuclear weapons program.
According to four sources familiar with the incident, the U.S. military moved to put plans in place to intercept the ship. Two of the sources noted that armed personnel from the U.S. military were preparing to board the ship, while military aircraft were flying overhead, according to one of these sources and another source familiar with the situation.
Just prior to the scheduled operation, officials scrutinized the report - which had been prepared by an analyst from the Special Operations Command - and discovered that it was produced with the assistance of artificial intelligence, and that the chatbot used by the analyst had incorrectly identified the nature of the materials the ship was carrying.
CNN was unable to ascertain what the incorrectly identified cargo was.
According to one source, the report was "completely wrong," but it "almost ignited a war" as any U.S. operation against a Chinese vessel could risk slipping into an armed conflict between the two countries.
U.S. military and intelligence officials are seeking to integrate artificial intelligence into nearly every aspect of their work; ranging from analyzing the vast amounts of raw intelligence data collected by the United States and identifying targets for military strikes, to more routine applications such as budget management, logistics, and supply chains.
However, this incident highlights the serious risks associated with the use of this powerful and still not fully understood technology in target identification during wartime.
Analysts have long expressed concerns that reliance on weak or distorted data in AI systems could lead to catastrophic miscalculations; a type of misjudgment that could, for example, prompt the U.S. to fire on a Chinese ship based on inaccurate information.
In this specific case, the analyst queried a chatbot about some intelligence reports regarding the ship's cargo manifest, which were reports issued by the U.S. Special Operations Command in the Pacific based in Hawaii. It is unclear whether this chat program is commercially available or a product of the U.S. government.
In this regard, a former senior U.S. official familiar with the AI systems used by military and intelligence analysts stated, "The internal tools are mostly just copies of commercial software, but they are dressed up to look different."