Play MNIST For Me! User Studies on the Effects of Post-Hoc, Example-Based Explanations & Error Rates on Debugging a Deep Learning, Black-Box Classifier

Example-based explanations can paradoxically reduce error detection: When users received case-based explanations for a classifier's decisions, they were more likely to perceive misclassifications as correct, suggesting that explanations may create false confidence rather than improve understanding.

Error rates above 4% significantly damage user trust: The study found a clear threshold effect where classifier accuracy below 96% led users to view the system as less correct, less reasonable, and substantially less trustworthy—highlighting that even modest error rates have outsized impacts on user confidence.

Post-hoc explanations require careful design to avoid harm: The finding that explanations increased acceptance of incorrect predictions suggests that poorly designed explainability features could actually make AI systems less safe by helping users rationalize errors rather than catch them.

User studies are essential for evaluating XAI effectiveness: With 349 participants across two experiments, this research demonstrates that explanations must be empirically tested with real users, as intuitive assumptions about what makes AI "interpretable" may not match actual human behavior and comprehension.