Stress-Testing Model Specs Reveals Character Differences among Language Models

Model specifications contain internal contradictions: The research identified direct conflicts between principles in AI constitutions and ethical guidelines used to train major language models from companies like Anthropic, OpenAI, and Google, meaning models are sometimes given incompatible instructions about how to behave.

Current AI guidelines don't cover complex real-world situations: The study found "insufficient coverage of nuanced scenarios," indicating that model specifications often fail to address the messy, ambiguous situations people actually encounter where multiple legitimate values compete with each other.

Different AI models behave very differently when facing ethical tradeoffs: Testing twelve frontier language models revealed over 70,000 cases where models gave significantly different responses to the same ethical dilemmas, suggesting that current specifications don't provide clear enough guidance for consistent behavior across the AI industry.

Behavioral disagreement signals deeper specification problems: The research demonstrates that when models disagree sharply on how to handle value tradeoffs, it's a reliable indicator that the underlying model specifications have problems like ambiguous wording or unresolved conflicts that need to be fixed.

Stress-testing can systematically expose specification weaknesses: By deliberately creating scenarios that force models to choose between competing principles, researchers can automatically identify thousands of problematic cases in model specifications before deployment, offering a practical method for improving AI safety.