Some Issues in Predictive Ethics Modeling: An Annotated Contrast Set of "Moral Stories"

High accuracy scores can mask serious vulnerabilities: Models like Delphi achieved 99.8% accuracy on ethical judgments, but small changes to how situations were described (as few as 3-5 words) dropped accuracy to 51%, revealing that these models may not truly understand ethics but rather pattern-match on specific wording.

How you describe a moral problem matters enormously: The paper found that simply adding misleading context about social norms reduced accuracy to 98.8%, and including textual bias (language that hints at the "right" answer) dropped it to 77%, demonstrating that AI ethics models are highly sensitive to how dilemmas are framed rather than grasping underlying principles.

Current ethics AI systems have likely overfit to their training data: The dramatic performance drops suggest these models memorized specific patterns in their training datasets rather than learning generalizable moral reasoning, meaning they may fail unpredictably when encountering real-world ethical situations described differently.

Practical safeguards are needed before deploying ethics AI: The researchers recommend three concrete steps: carefully examining how social norms are represented in training data, training models to ask clarifying questions rather than make snap judgments, and screening input for hidden biases that might mislead the system.

Accuracy alone is insufficient for evaluating AI ethics tools: This paper argues that developers must test how robust their models are to different descriptions of the same situation, not just measure overall accuracy scores.