Explanation Strategies for Image Classification in Humans vs. Current Explainable AI

Humans use different explanation strategies than AI systems: When explaining image classifications, humans adopted more exploratory visual scanning patterns rather than the focused attention they used for the actual classification task, suggesting people explain differently than they decide.

Two distinct human explanation approaches exist: Some people focused on foreground objects with conceptual explanations (e.g., "this is a dog because of the ears"), while others used broader visual exploration with more visual descriptions, and both approaches were effective for different reasons.

Current AI explanations align better with exploratory human strategies: Saliency maps from XAI systems showed the highest similarity to humans' exploratory scanning patterns, suggesting these AI explanations may feel more natural and accessible to users than other explanation methods.

AI explanations based on "observable causality" match human reasoning better: XAI methods that highlight features by showing what happens when you remove or change them (perturbation-based) aligned more closely with human explanation strategies than methods that simply point to internal features the model uses for decision-making.

Designing AI explanations around human diversity matters: Since humans use varied explanation strategies, XAI systems designed to match these natural human approaches—particularly those emphasizing cause-and-effect relationships—may be more understandable and trustworthy to end users.