System Safety and Artificial Intelligence

AI safety requires end-to-end analysis beyond just the algorithm: The paper emphasizes that AI systems cannot be safeguarded through technical design choices on the model or algorithm alone. Instead, safety requires examining the entire system including context of use, affected stakeholders, and the institutional environment—a lesson drawn from decades of system safety experience in safety-critical domains.

Safety is a socio-technical property, not purely technical: The paper argues that safety and other values are "inherently socio-technical and emergent system properties" that require design and control measures across technical, social, and institutional components. This means responsible AI requires coordination between engineers, policymakers, and affected communities, not just better code.

New AI applications create new and underestimated hazards: The paper identifies that "new applications of AI across societal domains and public organizations and infrastructures come with new hazards, which lead to new forms of harm, both grave and pernicious." This suggests current safety frameworks may not adequately anticipate risks from AI in novel contexts like public services and infrastructure.

There is no consensus on diagnosing AI system hazards: The paper notes "the lack of consensus for diagnosing and eliminating new AI system hazards," indicating that organizations deploying AI systems may lack shared frameworks for identifying what could go wrong—a critical gap for preventing harm.

System safety field offers proven lessons for AI: The paper draws on decades of experience from safety-critical software systems (like aviation and industrial automation) to provide concrete tools and frameworks that can be adapted to manage AI system safety more effectively.