Convergent perils: luminaries against human ethics being outsourced to AI

Why in the News:

A group of Nobel laureates, Artificial Intelligence (AI) scientists, and religious leaders signed the “Rome Declaration for an Unarmed and Disarming Peace” on 16 July, calling for a treaty banning autonomous systems from nuclear launch decisions. The Declaration argues that introducing AI into nuclear command structures does not merely speed up decision-making, it removes the human hesitation that has so far prevented nuclear war.

What has historically prevented nuclear escalation, and why is it fragile?

1. Human hesitation as the real safeguard: Mutually Assured Destruction (MAD), the doctrine that a first nuclear strike triggers devastating retaliation, relied on human hesitation, diplomatic backchannels, and moral pause, not technical safeguards alone.
2. Historical near misses: During the 1962 Cuban Missile Crisis and the 1983 Soviet false alarm incident involving officer Stanislav Petrov, disaster was averted because human operators exercised caution and scepticism.
3. Compressed decision windows: AI-driven predictive logistics and battle management systems compress decision windows from hours to seconds, removing the time needed for human verification.

What technical failure modes make AI unsuitable for this role?

1. Documented failure modes: Large Language Models (LLMs) and neural networks are prone to hallucinations and data poisoning, where training data is maliciously corrupted to manipulate outputs.
2. Automatic escalation risk: In a crisis, an algorithm misreading a routine signal as an imminent threat could automatically trigger a retaliatory sequence within a compressed decision window.
3. Unauditable reasoning: Most AI models operate as “black boxes,” making their conclusions impossible to verify at the moment a launch decision would be required.

What five operational principles does the Declaration propose?

1. Mandatory meaningful human control: Prohibiting any AI-driven system from making the final decision on the use or deployment of nuclear weapons.
2. The digital commons model: Expanding data access so experts beyond major powers can better assess AI-related existential risks.
3. Responsible development: Requiring AI developers to publish the ethical frameworks guiding their models and forbidding fully autonomous, self-improving systems that cannot be monitored or halted by human operators.
4. Internal arsenal vulnerability audits: Requiring nuclear states to conduct rigorous reviews of their command-and-control systems against AI-driven cyber tampering.
5. Time-bound, verifiable disarmament: Renewing good faith international negotiations toward completely and verifiably eliminating nuclear weapons.

Why is the current Iran nuclear crisis treated as a live test of these risks?

1. Broken arms control architecture: Iran’s breaches of its non-proliferation obligations, followed by joint United States-Israel military strikes, have exposed the existing arms control architecture as broken.
2. AI in contested monitoring: States increasingly use AI for target characterisation and predictive intelligence tracking of nuclear facilities under contested conditions.
3. Automated misjudgment risk: An AI system defending an enrichment facility could misinterpret a routine network probe or a physical strike as the start of an attack, triggering a disproportionate automated response before any human decision is made.

What obstacle remains to acting on the Declaration’s warning?

1. No treaty consensus: No consensus currently exists among states on what a treaty banning AI in nuclear command should cover, or even on banning autonomous AI in warfare more broadly.
2. Moral instrument, not binding law: The Declaration functions as a moral and philosophical lexicon rather than a binding legal instrument at this stage.
3. Dependence on public pressure: Its success depends on generating public awareness that then creates grassroots pressure for states to negotiate a binding treaty.

Conclusion:

The Declaration’s central claim is that no technical safeguard can substitute for the human doubt that has twice averted nuclear war, and that compressing decision windows to seconds through AI removes exactly that safeguard. No binding treaty yet exists to prevent this, and the current Iran nuclear crisis shows the danger is not hypothetical. Ethical responsibility for nuclear launch decisions cannot be delegated to a machine.