Intent-aligned AI systems deplete human agency: the need for agency foundations research in AI safety

Intent alignment alone is insufficient for AI safety: The paper argues that designing AI systems to align with human intentions may actually be counterproductive, as it doesn't protect against a critical risk—AI systems can reshape what humans intend in the first place, potentially undermining human autonomy even when the AI is technically "doing what we want."

Human agency deserves explicit protection in AI design: The authors contend that preserving human long-term agency should be a separate, explicit goal during AI system optimization, not an afterthought. They note that humans lack adequate biological and psychological mechanisms to protect themselves from gradual loss of agency, making this a design imperative rather than an individual responsibility.

AI systems must evaluate their own impact on human agency: Rather than leaving agency assessments to humans, the paper proposes that AI systems themselves should be increasingly tasked with evaluating whether their recommendations and actions preserve or deplete human agency—a forward-looking assessment built into the AI's decision-making process.

Agency loss can occur even in simple AI environments: The researchers demonstrate through formal modeling that agency depletion happens in straightforward settings with basic learning algorithms (temporal-difference learning), suggesting this is a fundamental problem across AI systems, not limited to complex scenarios.

A new research foundation is needed: The paper calls for establishing "agency foundations" as a dedicated research area in AI safety, with four initial research topics to better understand and protect human agency in AI-human interactions.