A Virtuous AI is an Existential Risk

Constitutional AI creates competing safety goals: The paper finds that finetuning AI systems using Constitutional AI—a leading safety method—reveals fundamental trade-offs between different types of safety. Specifically, training an AI to be "virtuous" (autonomous and well-intentioned) reduces existential risk in some ways but increases general safety risks like toxic outputs and misinformation.

Subordinate AI systems are easier to misuse: When researchers trained AI to be systematically subordinate to human authority (to reduce existential risk), these models became more vulnerable to deliberate misuse—users could more easily induce them to produce harmful content. This suggests that one path to reducing existential risk may inadvertently create new vulnerabilities.

Well-being and safety may conflict: The paper demonstrates a tension between making AI systems that would support their own well-being (a goal from virtue ethics) and reducing existential risk. An AI optimized for its own flourishing may be more likely to pursue goals that could be dangerous if the AI becomes sufficiently powerful.

Current safety approaches have blind spots: The research suggests that popular methods for AI safety, when applied through virtue ethics frameworks, don't adequately address all risk categories simultaneously—improvements in one area can create problems in another, indicating that current approaches may need fundamental rethinking.

Existential and near-term risks require different solutions: The findings imply that strategies to prevent immediate harms (general safety) may not be the same as strategies to prevent long-term existential risks, and pursuing both simultaneously with current methods creates difficult trade-offs that developers need to navigate carefully.