Intolerable Risk Threshold Recommendations for Artificial Intelligence

AI companies are committing to identify "red lines" for safety risks: Following the AI Seoul Summit in May 2024, 16 major AI organizations and 27 nations agreed to define specific thresholds where AI risks become unacceptable, marking a significant step toward industry-wide safety standards and government oversight.

Eight major risk categories require threshold-setting, including weapons development (CBRN), cyberattacks, autonomous systems, manipulation, deception, toxicity, discrimination, and economic disruption—reflecting the diverse ways frontier AI models could cause harm across society.

Perfect safety thresholds are unrealistic given rapid AI advancement: The paper recommends aiming for "good, not perfect" thresholds because AI capabilities are evolving faster than our ability to fully understand and measure associated risks, requiring pragmatic rather than idealistic standards.

Thresholds must address both intentional misuse and unintended failures: The risks aren't limited to bad actors deliberately weaponizing AI; the paper emphasizes that system failures, cascading effects, and simultaneous failures across multiple models pose equally serious threats requiring preventive measures.

This framework provides concrete starting points for implementation: By offering specific case studies and recommendations across eight risk categories, the paper gives organizations and governments actionable guidance for operationalizing safety commitments rather than leaving them as vague aspirations.