Does Explanation Correctness Matter? Linking Computational XAI Evaluation to Human Understanding

Explanation correctness matters, but not uniformly: The study found that explanation accuracy only significantly impacted human understanding at lower levels (70% and 55%), with no additional decline below 70%. This suggests there may be a threshold effect rather than a linear relationship between explanation quality and comprehension.

Perfect explanations don't guarantee understanding: Even when explanations were 100% correct, participants showed bimodal performance—only some achieved strong understanding while others performed near chance level. This indicates that explanation correctness alone is insufficient for ensuring human comprehension of AI decisions.

Correctness affects who understands, not just how well: Lower explanation correctness reduced the proportion of participants who could accurately predict the AI's decisions rather than uniformly lowering everyone's accuracy. This suggests that poor explanations may help some people while leaving others unable to understand at all.

Current XAI evaluation metrics may be incomplete: The paper challenges the common assumption that higher "faithfulness" or "fidelity" scores automatically lead to better human understanding. Organizations relying solely on computational correctness metrics to evaluate explanations may be missing critical gaps in actual user comprehension.

Individual differences in explanation comprehension are significant: Some participants accurately predicted AI decisions but described incorrect reasoning patterns, suggesting that people process explanations differently. This highlights the need for user-centered evaluation beyond just measuring explanation accuracy.