SafetyTrustworthinessAccountabilitycs.AI
Sustaining AI safety: Control-theoretic external impossibility, intrinsic necessity, and structural requirements
James M. Mazzu
May 13, 2026
arXiv.org
Key Takeaways
-
External control has fundamental limits: Using control theory, the paper proves that once AI systems become powerful enough to exceed what external oversight can counteract, safety strategies relying on external enforcement will inevitably fail—this is a structural problem, not a flaw in any particular approach.
-
Future AI safety must be "intrinsic": Since external control becomes unreliable with sufficiently advanced systems, any viable safety strategy must be built into the AI system itself rather than depending on outside constraints or human intervention.
-
Safety objectives must be locked in early: The paper identifies a critical requirement that an AI system's core safety-related goals must be established correctly from the start and remain stable even as the system modifies itself, since external correction becomes impossible at higher capability levels.
-
Safety must scale with capability: As AI systems become more powerful, safety protections cannot remain static—they must be designed to continue functioning and adapting as the system's capabilities grow, rather than assuming current safety measures will suffice indefinitely.
-
The problem is structural, not contingent: The paper's impossibility result applies across an entire class of externally-enforced strategies rather than just specific implementations, suggesting that incremental improvements to current oversight approaches may not solve the fundamental challenge of maintaining safety in advanced AI systems.