Are language models rational? The case of coherence norms and belief revision

Language models vary in their adherence to basic rationality standards: The paper finds that some language models follow coherence norms (logical consistency and proportional belief strength), while others do not—suggesting inconsistent levels of rational behavior across different models, which has implications for predicting and controlling their outputs.

A new framework for measuring "belief strength" in AI: The researchers propose using a model's internal probability scores for predicting the next word as a measure of how strongly it "believes" something, introducing the Minimal Assent Connection (MAC). This provides a concrete way to assess whether language models maintain consistent confidence levels across related statements.

Rationality matters for AI safety and alignment: The paper argues that whether language models behave rationally is directly relevant to AI safety concerns, since rational behavior is more predictable and explainable—qualities essential for ensuring AI systems behave as intended and can be properly monitored.

Current language models may not be reliably coherent: The finding that some language models fail basic coherence tests (like maintaining logical consistency or proportional confidence across related beliefs) suggests these systems could produce contradictory or unreliable outputs, even when they appear fluent and confident.

Rationality assessment should be part of model evaluation: The paper implies that testing for coherence and rational behavior should become a standard part of evaluating language models before deployment, similar to how we might assess other critical system properties.