How to Answer Why -- Evaluating the Explanations of AI Through Mental Model Analysis

Users develop mental models of AI systems based on their interactions, and when AI systems don't provide explicit explanations, these models are formed primarily through observation and experience rather than accurate understanding of how the system actually works.

The accuracy of users' mental models directly impacts their decisions about whether and how to use AI systems—if a user's understanding of how an AI works doesn't match reality, they may make poor choices about trusting or relying on that system.

Current AI explanations may not be effectively teaching users how systems actually function; the paper identifies a critical gap between what users think AI does and what it actually does, suggesting many existing explanations fall short of their intended purpose.

Mental model analysis is proposed as a practical evaluation method for testing whether explainable AI actually works—rather than just assuming explanations help, researchers can directly measure whether users develop accurate understandings of AI systems.

Responsible AI development requires human-centered testing approaches that go beyond technical metrics to verify that users can form valid mental models, ensuring they have the knowledge needed to make informed decisions about AI system use.