A Psychological Theory of Explainability

Humans judge AI explanations by comparing them to their own reasoning: The paper finds that people interpret AI explanations (specifically saliency maps showing which parts of an image the AI focused on) by comparing them to explanations they would naturally give themselves. This means an explanation is only effective if it aligns with human intuition about how someone should solve that problem.

Current XAI validation is inefficient and unsystematic: Without a precise theory of how humans understand explanations, companies and researchers must test each AI explanation individually through trial-and-error. The paper argues this case-by-case approach prevents the field from building general principles about what makes explanations work across different AI systems and contexts.

People assume AI thinks like them unless told otherwise: The research reveals that without an explanation, humans expect AI systems to make decisions the same way they would. This baseline assumption shapes how people interpret the explanations they receive, which has important implications for how AI developers should design and present their systems.

Psychological theory can predict how people will react to AI explanations: By applying Shepard's universal law of generalization—a established cognitive science principle—the authors created a mathematical model that accurately predicted what study participants would conclude from saliency map explanations, suggesting that human interpretation of AI explanations follows predictable psychological patterns.