CoAX: Cognitive-Oriented Attribution eXplanation User Model of Human Understanding of AI Explanations

XAI explanations aren't automatically helping users understand AI decisions. Despite innovations in explainable AI methods, recent user evaluations show that the goal of improving user understanding remains unmet, suggesting current approaches may not align with how people actually think.

Different explanation types (feature importance, feature attribution) lead people to use different reasoning strategies. The research identified that users employ varying cognitive approaches depending on which XAI method they receive, meaning a one-size-fits-all explanation approach won't work for all users.

Cognitive models of human reasoning outperform standard machine learning approaches at predicting user decisions. By modeling the actual mental processes people use when interpreting explanations, the researchers created better predictors of how users will act on AI explanations than traditional ML proxies could achieve.

Understanding why users struggle with explanations requires studying human cognition, not just improving the explanations themselves. The paper demonstrates that debugging XAI requires examining the cognitive processes underlying human decision-making, not just iterating on explanation designs.

Cognitive models can reduce the need for expensive user studies in XAI research. Once fitted to human behavior, these models can be used to test hypotheses and investigate research questions without repeatedly recruiting and testing real participants, making XAI research more efficient and scalable.