AI Safety: A Climb To Armageddon?

Safety measures may inadvertently increase risk if they enable more powerful AI systems: The paper argues that under certain conditions, safety efforts could have "negative expected utility" because they may allow AI systems to become more powerful before they fail, potentially amplifying harm when failures occur.

Three fundamental challenges complicate AI safety efforts: The paper identifies "Bottlenecking" (constraints on safety implementation), the "Perfection Barrier" (the impossibility of perfect safety), and "Equilibrium Fluctuation" (instability in safety conditions) as intrinsic features that undermine traditional safety approaches.

The core tension: power versus safety timing: The paper's central claim rests on the assumption that "the expected correlation between an AI system's power at the point of failure and the severity of the resulting harm" means that making systems safer while simultaneously making them more capable could backfire if failure remains inevitable.

Current safety strategies face structural limitations: The paper examines three response approaches—Optimism, Mitigation, and Holism—and argues each faces inherent challenges, suggesting that conventional wisdom about AI safety may need fundamental reconsideration.

This work calls for re-examination of basic AI safety assumptions: Rather than providing solutions, the paper's "surprising robustness" suggests the field should question foundational premises about how safety measures interact with AI development trajectories.