War machines can run amok with AI in control

AIs not only make life-and-death targeting decisions on the battlefield, but also involve humans in only some stages of the decision-making process. To illustrate how that happens and prompt greater scrutiny of model-enabled killing, Airwars, a not-for-profit transparency watchdog, has published a detailed report, taking apart the ways machine learning powers modern warfare. The investigation, titled Anatomy of an AI Kill Chain, offers a visual guide through a fictitious kill chain – the steps real militaries go through to identify targets and eliminate them, with an emphasis on decisions delegated to AI.

Authors Sophia Goodfriend, Heidy Khlaaf, Namir Shabibi, Joe Dyke, and Nathan Walker cover six stages: the decision support systems used to assist data gathering, surveillance technology, intelligence and identification, target selection, strikes on targets, and post-strike assessments. The report notes, "A recent book about US military AI revealed that in some operations only two of the six stages of the US military kill chain now involved humans in the loop, with a third involving some human oversight. The rest are now fully automated."

Given that targeting mistakes have been widely documented, it's worth wondering whether AI is making such errors more common. And since military officials have, in the past, insisted AI's role cannot be known, there's reason to look for ways to better understand when AI is involved in life-and-death decisions. "In journalism and in policy, there's a tendency to focus on maybe one autonomous weapon system, like an Anduril drone or a Palantir anomaly detection system, and to look at the specific companies that are producing a few single machine learning algorithms that are undergirding those systems," said Sophia Goodfriend, research fellow at the University of Cambridge’s Pembroke College and co-author of the report, in a phone interview with The Register.

"What we wanted to do … is to really underscore the stack of AI systems that