The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems

Most AI safety risks are "quiet" rather than dramatic: The paper argues that current safety discussions focus too heavily on obvious harms and catastrophic scenarios, while the most consequential failures in deployed systems are often subtle, distributed across multiple components, and gradually normalized into workflows before being recognized as problems.

The real safety challenge is maintaining visibility and contestability: Beyond preventing harmful outputs, the critical issue is whether socio-technical systems preserve conditions where errors remain visible, can be questioned, can be contained, and can be recovered from—suggesting that transparency and human oversight mechanisms are as important as model safety itself.

Five interconnected integrity layers require attention: The paper identifies epistemic integrity (honest uncertainty representation), control integrity (robust authority and permissions), temporal integrity (safety across time and updates), organizational integrity (institutional capacity to audit and intervene), and ecosystem integrity (preserving information environments for oversight)—indicating that AI safety requires attention across technical, organizational, and societal dimensions.

Current safety instrumentation has significant blind spots: The framework highlights that existing safety measures may not adequately monitor these "hidden" failure modes, suggesting that organizations deploying AI systems may have false confidence in their safety practices if they only measure obvious harms.

Institutional capacity for oversight is itself at risk: The paper emphasizes that as AI systems become more complex and integrated into workflows, organizations' ability to audit, assign responsibility, and intervene effectively may erode—a meta-level safety concern that requires proactive institutional design.