From nuclear safety to LLM security: Applying non-probabilistic risk management strategies to build safe and secure LLM-powered systems

Traditional risk analysis methods fall short for AI systems: The paper argues that conventional probabilistic risk analysis—which requires identifying and quantifying all possible risks—is impractical for LLMs because these systems are novel, complex, and face adaptive adversaries who continuously find new attack methods. This means we cannot simply apply the risk management approaches that worked for nuclear power plants or bridges.

Over 100 proven strategies from other engineering fields can be adapted for LLM safety: Rather than starting from scratch, the authors demonstrate that established risk management techniques from nuclear safety, civil engineering, and other fields—such as event tree analysis and robust design principles—can be mapped and applied to LLM security challenges, offering a practical toolkit for developers.

Non-probabilistic approaches are more suitable than statistical prediction: Because LLM risks are difficult to predict statistically, the paper recommends focusing on non-probabilistic strategies (like designing systems to be inherently robust or building in multiple layers of protection) rather than trying to calculate the probability of specific failures occurring.

A structured framework can guide responsible AI implementation: The authors present an LLM-powered workflow and categorize strategies into five groups, providing solution architects with a systematic approach to identify and implement safety measures—moving responsible AI from abstract principles to concrete, actionable steps.

Adaptive adversaries require ongoing, dynamic defenses: The paper emphasizes that because attackers continuously evolve their methods, static risk management is insufficient; organizations need strategies designed to anticipate and adapt to novel threats rather than only protecting against known vulnerabilities.