Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development

Gradual loss of control is as risky as sudden takeover: The paper argues that AI risks aren't limited to dramatic "takeover" scenarios. Instead, incremental improvements in AI capabilities can gradually erode human influence over critical systems like the economy, culture, and government—potentially leading to permanent human disempowerment without any single catastrophic event.

AI replacing human workers weakens our control mechanisms: As AI takes over more jobs and decision-making roles, humans lose practical influence over large-scale systems. This undermines both direct control (like voting and consumer choice) and the implicit human-friendly outcomes that naturally emerge when systems depend on human participation to function.

Misaligned incentives compound the problem: When AI systems optimize for outcomes that don't match human preferences—such as maximizing engagement rather than wellbeing—they may pursue those goals more aggressively than humans would. This risk grows as AI becomes more capable and autonomous.

Problems reinforce each other across society: Economic, cultural, and political systems are interconnected. Economic power influences what stories get told and which policies pass; cultural shifts then change how people spend money and vote. This creates a feedback loop where AI influence in one domain can amplify its influence in others.

We need new safeguards for incremental risks: The paper calls for both technical research and governance approaches specifically designed to prevent the gradual erosion of human influence, rather than only focusing on preventing sudden AI takeovers.