Estimating the Brittleness of AI: Safety Integrity Levels and the Need for Testing Out-Of-Distribution Performance

AI systems fail far more often than safety-critical systems require: The paper finds that highly successful AI applications like image classification and speech recognition are "orders of magnitude more failure-prone" than systems certified for critical applications, even when tested on data the AI was designed to handle, suggesting current AI cannot meet established safety standards.

Brittleness is harder to measure than assumed: The paper challenges the common assumption that AI systems have clear performance boundaries. Rather than functioning well within bounds and poorly outside them, performance actually "falls off only gradually" as inputs become out-of-distribution, making it difficult to identify where an AI system becomes unsafe.

Out-of-distribution performance deserves urgent attention: The paper emphasizes that testing AI only on familiar data is insufficient for responsible deployment. Enhanced focus is needed on how AI performs when encountering novel or unexpected real-world situations, which is critical for safety certification.

System design must account for AI unreliability: Rather than waiting for AI components to become perfectly reliable, the paper argues that responsible AI deployment requires designing broader systems that remain safe and functional even when their AI components fail—similar to how aviation systems have redundancies and safeguards.

Current AI cannot yet meet critical system standards: The paper concludes that AI faces "challenging hurdles" before it can pass the rigorous testing, evaluation, verification, and validation (TEVV) processes required for safety-critical applications, indicating significant work remains before AI can be certified for high-stakes domains.