Model-Based Soft Maximization of Suitable Metrics of Long-Term Human Power

Power balance matters for AI safety: The paper argues that managing power distribution between humans and AI systems—rather than allowing AI to accumulate unchecked power or strip humans of agency—is a key safety consideration alongside traditional utility maximization approaches.

Human wellbeing requires preserving choice: The research emphasizes that human power (defined as "the ability to pursue diverse goals") is essential for wellbeing, suggesting that AI systems designed to maintain or expand human options may better serve human flourishing than systems optimized for narrow objectives.

A new safety metric has been developed: The authors propose a specific, measurable objective function for AI systems that explicitly maximizes "long-term aggregate human power" while accounting for inequality, risk, human limitations, and social norms—offering a concrete alternative to traditional reward-based AI objectives.

"Soft maximization" may be safer than direct optimization: The paper cautiously suggests that gently encouraging AI systems toward human empowerment metrics could be safer than hard-coded utility functions, potentially reducing risks from both sudden disempowerment and gradual erosion of human agency.

This approach addresses multiple governance levels: By framing power balance as a core objective, the framework potentially applies to individual AI agents, human-AI teams, and international AI governance—suggesting a unified principle for responsible AI development across different scales.