SafetyAccountabilityTrustworthinesscs.CY
Managing extreme AI risks amid rapid progress
Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Trevor Darrell, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, Gillian Hadfield, Jeff Clune, Tegan Maharaj, Frank Hutter, Atılım Güneş Baydin, Sheila McIlraith, Qiqi Gao, Ashwin Acharya, David Krueger, Anca Dragan, Philip Torr, Stuart Russell, Daniel Kahneman, Jan Brauner, Sören Mindermann
October 26, 2023
arXiv.org
88 citations
1 influential
Key Takeaways
-
AI systems are becoming more autonomous and goal-directed, moving beyond narrow, task-specific applications toward "generalist AI systems that can autonomously act and pursue goals," which could dramatically amplify both benefits and risks at scale.
-
Current safety efforts are insufficient relative to the pace of AI development—the paper emphasizes that "AI safety research is lagging" and "society's response...is incommensurate with the possibility of rapid, transformative progress that is expected by many experts."
-
Existing governance frameworks have critical gaps, particularly lacking "mechanisms and institutions to prevent misuse and recklessness" and failing to adequately address autonomous systems, despite some promising initial policy steps.
-
Extreme risks include irreversible loss of human control, alongside large-scale social harms and malicious uses—the paper warns of "an irreversible loss of human control over autonomous AI systems" as a key concern requiring urgent attention.
-
A dual approach combining technical and governance solutions is necessary, drawing lessons from other safety-critical technologies (like aviation or nuclear energy) to develop both better AI safety research and proactive, adaptive governance mechanisms that can keep pace with rapid AI progress.