Diagnosing AI Explanation Methods with Folk Concepts of Behavior

Explanations fail when they don't match how humans naturally understand behavior. The paper argues that AI explanation methods are often unsuccessful not because they lack information, but because they don't communicate in the "folk concepts" (everyday mental models) that humans naturally use to understand why others behave a certain way—such as goals, beliefs, and social norms.

Current XAI methods have identifiable gaps in what they communicate. By mapping popular explanation techniques to folk concepts of behavior, the researchers found that many existing methods are missing key information constructs. For example, an explanation might tell you what a system did but fail to convey why it did it in terms humans intuitively grasp, leading to misunderstandings about AI behavior.

A narrative-based framework can improve AI explanations. The paper proposes using an explanatory narrative blueprint (shown in Figure 1) that structures AI explanations around folk concepts of behavior, suggesting this approach could help people better understand and appropriately trust AI systems.

Responsible AI requires designing explanations for human comprehension, not just technical accuracy. The research highlights that transparency in AI isn't just about providing information—it's about ensuring that information is actually understood by the people who need to make decisions about or with AI systems.