SafetyAccountabilityTrustworthinesscs.LG
Landscape of AI safety concerns -- A methodology to support safety assurance for AI-based autonomous systems
Ronald Schnitzer, Lennart Kilian, Simon Roessner, Konstantinos Theodorou, Sonja Zillner
December 18, 2024
International Conference on System Reliability and Safety
6 citations
1 influential
Key Takeaways
-
AI systems in autonomous applications face unique safety challenges that differ from traditional software because AI components often lack concrete specifications and exhibit uncertain behavior that is difficult to predict, making it harder to prove a system is safe.
-
A systematic approach to identifying AI safety concerns is necessary: The paper proposes the "Landscape of AI Safety Concerns" methodology, which works by systematically demonstrating the absence of known AI safety problems rather than trying to guarantee perfect performance—a more practical approach given AI's inherent uncertainties.
-
Safety assurance for AI requires analyzing both the system and its environment: The complexity of both how AI systems work internally and the unpredictable real-world conditions they operate in (demonstrated through a driverless train case study) means safety cases must account for multiple sources of uncertainty simultaneously.
-
Thorough documentation of AI-specific risks strengthens safety cases: By carefully analyzing and mitigating AI-specific insufficiencies, organizations can build more convincing evidence that autonomous systems are safe enough to deploy, which is essential for public trust and regulatory approval.
Abstract
Artificial Intelligence (AI) has emerged as a key technology, driving advancements across a range of applications. Its integration into modern autonomous systems requires assuring safety. However, the challenge of assuring safety in systems that incorporate AI components is substantial. The lack of concrete specifications, and also the complexity of both the operational environment and the system itself, leads to various aspects of uncertain behavior and complicates the derivation of convincing evidence for system safety. Nonetheless, scholars proposed to thoroughly analyze and mitigate AI-specific insufficiencies, so-called AI safety concerns, which yields essential evidence supporting a convincing assurance case. In this paper, we build upon this idea and propose the so-called Landscape of AI Safety Concerns, a novel methodology designed to support the creation of safety assurance cases for AI-based systems by systematically demonstrating the absence of AI safety concerns. The methodology's application is illustrated through a case study involving a driverless regional train, demonstrating its practicality and effectiveness.