ExplainabilityTransparencyAccountabilitycs.LG
Counterfactual Explanations for Misclassified Images: How Human and Machine Explanations Differ
Eoin Delaney, Arjun Pakrashi, Derek Greene, Mark T. Keane
December 16, 2022
Artificial Intelligence
34 citations
1 influential
Key Takeaways
-
Most counterfactual explanation methods haven't been tested with real users: Only about 7% of the 100+ existing counterfactual explanation methods have been validated through user studies, meaning the field has largely assumed these explanations work without actually confirming this with people.
-
AI and humans generate fundamentally different explanations: When shown misclassified images, humans don't make small, minimal changes like most AI algorithms do. Instead, they make larger, more meaningful edits that better represent what a correct example of the other category would actually look like.
-
Current AI explanations may not match human intuition: The disconnect between how machines and humans explain image misclassifications suggests that popular counterfactual explanation methods may not be as psychologically valid or user-friendly as previously claimed, potentially limiting their effectiveness for helping people understand AI decisions.
-
"Meaningful edits" better reflect real-world reasoning: Humans appear to think in terms of prototypes—what a typical example of the correct category looks like—rather than finding the minimal path to change a classification, suggesting AI explanations should be redesigned to align with this more intuitive approach.
Abstract
Counterfactual explanations have emerged as a popular solution for the eXplainable AI (XAI) problem of elucidating the predictions of black-box deep-learning systems due to their psychological validity, flexibility across problem domains and proposed legal compliance. While over 100 counterfactual methods exist, claiming to generate plausible explanations akin to those preferred by people, few have actually been tested on users ($\sim7\%$). So, the psychological validity of these counterfactual algorithms for effective XAI for image data is not established. This issue is addressed here using a novel methodology that (i) gathers ground truth human-generated counterfactual explanations for misclassified images, in two user studies and, then, (ii) compares these human-generated ground-truth explanations to computationally-generated explanations for the same misclassifications. Results indicate that humans do not "minimally edit" images when generating counterfactual explanations. Instead, they make larger, "meaningful" edits that better approximate prototypes in the counterfactual class.