‘A moral red line’? World on cusp of autonomous weapons revolution

Lethal autonomous weapons powered by artificial intelligence are set to revolutionise warfare but there is no proper regulatory or legal framework governing the new technology. Read Also: 6,000 Flight Cancelled: US air travel resumption will ‘take time’ A conference starting Monday at the United Nations in Geneva will discuss how to establish curbs and prohibitions on such weapons, called killer robots by some.

But some analysts doubt the discussions at the Convention on Certain Conventional Weapons (CCW) will keep pace with the fast-changing weapons. AFP looks at three key questions ahead of the conference. What are LAWS? Lethal Autonomous Weapons Systems (LAWS) “can identify, select, and engage a target without the intervention of a human operator in the execution of these tasks,” said Jean-Marc Rickli, head of global and emerging risks at the Geneva Centre for Security Policy.

Autonomous weapons have long existed. An anti-personnel mine “is an autonomous weapon in the sense that it explodes when someone steps on it,” said Laure de Roucy-Rochegonde of the Centre for Geopolitics of Technology at the French Institute of International Relations (IFRI). But with the proliferation of AI-enhanced drones, such weapons are now “capable of selecting and engaging a target that was not predetermined, adapting to their environment without human intervention,” said Rickli.

So the critical issue for potential regulators lies in determining the level of human control over the decision to employ such a weapon. The United States refers to an “appropriate levels of human judgement” in the use of autonomous weapons. France maintains that a human must remain in the decision-making loop and bans “lethal autonomous weapons systems” (LAWS) but permits “partially autonomous” systems known as PLAWS.

“The human role is always ensured through what is termed ‘appropriate control’ by a military commander — a concept that varies in scope depending on the system’s criticality and the operational context,” said Rear Admiral Vincent Sebastien, deputy director of the French defence ministry’s Agency for Defence Artificial Intelligence (AMIAD). The ethical issues?

“We are now dangerously close to crossing a moral red line: the autonomous targeting of humans by machines,” UN Secretary General Antonio Guterres, and International Committee of the Red Cross president Mirjana Spoljaric warned in a joint statement in August. But analysts said the Geneva discussions, while important, will touch only part of the issue.

Today, the diminishing role of humans is being driven by the growing use of AI in analysis and targeting. Rickli said the scope of the agenda in Geneva is “very narrow”. “It fails to encompass the full range of issues surrounding AI on the battlefield” – an area where its use is expanding, particularly in target generation. He said this was seen in the US-Israeli war against Iran this year “where the Americans engaged as many as 1,000 targets per day” inside Iran.

“There are a host of problems and unacceptable risks associated with targeting and decision-support systems,” added de Roucy-Rochegonde. “Part of this debate concerns systems that are already widely deployed on the battlefield.” “What kind of human supervision is required to ensure that the operator is genuinely exercising their own judgement rather than simply providing a rubber stamp for the machine’s recommendations?” she asked.

How quickly is the situation changing? The autonomous weapons to be discussed in Geneva are not yet in widespread use, although analysts have noted that they have been deployed by Russia in its invasion of Ukraine as well as US attacks in Iran. But the spread is inevitable. “Autonomy will very quickly become a reality on the battlefield,” warned rear admiral Sebastien.

“Autonomous weapon systems are not part of a distant future – a race towards greater autonomy in weapons systems is already well underway,” Guterres and Spoljaric said in their statement. Yet significant doubts and dangers remain, Dario Amodei, head of the AI giant Anthropic, warned this year: “Today, frontier AI systems are simply not reliable enough to power fully autonomous weapons.” But ultimately, the incentive for major powers is too strong.

Swarms of autonomous drones – which “aim to neutralise an adversary through saturation” – would be highly effective, and some states view them as a crucial strategic advantage, said Rickli. In January, the Chinese defence ministry described swarms of autonomous drones as the “new face of warfare.” “In an era of sharp military competition, the first state to normalise fully autonomous lethal strikes will reset expectations for everyone else,” said Gerald Mako, a research affiliate of Cambridge University in the Small Wars Journal.