Where AI Assurance Might Go Wrong: Initial lessons from engineering of critical systems

System boundaries for AI are too narrow: The paper identifies a critical gap where AI safety frameworks fail to adequately define what constitutes "the system" being evaluated. In traditional critical systems like aircraft, boundaries are clearly established, but AI systems often lack this clarity, potentially leaving important risks unexamined.

Everyday safety methods cannot scale to critical AI systems: The authors emphasize "orders of magnitude difference in confidence needed in critical rather than everyday systems," warning that standard AI development and testing practices are fundamentally insufficient for high-stakes applications and cannot simply be made more rigorous without rethinking their foundations.

Risk tolerability must be explicitly defined before development begins: The paper stresses that current AI safety frameworks insufficiently elaborate "the tolerability and nature of the risks," meaning organizations often fail to clearly establish acceptable risk levels upfront—a foundational step in traditional critical systems engineering that must be adopted for AI.

Assurance methods lack adequate theoretical grounding: A core concern raised is that current AI assurance approaches "lack theories that would allow behaviours to be adequately assured," meaning we cannot reliably predict or verify how AI systems will behave in real-world conditions the way engineers can with aircraft systems.

Assurance cases should evaluate both system and decision criticality: The authors advocate for "Assurance 2.0" approaches that assess not just whether an AI system is safe enough, but whether the decision-making process around deploying that system appropriately accounts for how critical that decision is to society.