Are explainable AI (XAI) evaluation strategies aligned? Comparing subjective, objective, and mathematical evaluation measures using saliency maps

Different evaluation methods can reach contradictory conclusions about the same AI explanation tool. The study tested three popular explanation techniques (LIME, Grad-CAM, and Guided Backpropagation) and found that subjective measures (what users reported feeling), objective measures (how well users actually performed tasks), and mathematical metrics (technical quality scores) each favored different methods—suggesting that picking one evaluation approach alone could lead to misleading conclusions about which explanations work best.

Mathematical quality scores don't reliably predict whether explanations actually help people. The researchers found that mathematical metrics measuring explanation "faithfulness" were only partially related to user task performance, and sometimes showed counterintuitive relationships. This means a technically sophisticated explanation score doesn't guarantee the explanation will be useful to real people making decisions.

User trust and satisfaction may not reflect explanation quality. Participants reported similar levels of trust and satisfaction across all three explanation methods, even though the methods performed differently on objective tasks and mathematical metrics. This suggests people's subjective feelings about an AI system's explanations may not be a reliable indicator of whether those explanations are actually good.

Comprehensive XAI evaluation requires multiple measurement approaches. Rather than relying on any single evaluation method, responsible AI development should combine subjective feedback, real-world task performance, and mathematical metrics to get a complete picture of whether explanations are truly helpful and trustworthy.