When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment

AI systems need to understand when humans break their own rules: The paper identifies a critical gap in AI safety—current language models struggle to recognize that humans don't always follow moral rules rigidly, especially in novel or unusual situations. This flexibility is essential for AI systems that will work alongside humans in real-world scenarios.

A new approach combining AI and cognitive science improves moral reasoning: The researchers developed "Moral Chain of Thought" (MORALCOT), which incorporates theories from moral psychology into how language models process ethical questions. This method outperformed seven existing language models by 6.2% F1 score, demonstrating that grounding AI in human reasoning research produces measurable improvements.

Current AI systems have significant blind spots in moral judgment: The paper presents a new dataset of rule-breaking scenarios (RBQA) that reveals where state-of-the-art language models fail to predict human moral decisions. This suggests that deploying these systems in high-stakes domains without addressing these gaps could lead to misaligned or unsafe outcomes.

Understanding context and exceptions is crucial for AI safety: The research emphasizes that truly safe AI requires systems that can recognize contextual factors that justify breaking rules—not just memorizing rules themselves. This is particularly important as AI systems take on more decision-making roles in human-centered domains.