From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI

Most XAI papers lack rigorous evaluation: One in three papers introducing explainability methods rely solely on anecdotal evidence (cherry-picked examples) rather than systematic testing, making it difficult to trust claims about whether explanations actually work.

User testing is rare in XAI research: Only one in five papers evaluate their explanations with actual users, despite the goal of XAI being to help people understand AI systems—this gap between developer assumptions and real-world usability is a significant concern.

Explainability is multifaceted, not binary: Rather than simply asking "is this explainable or not?", the paper identifies 12 distinct properties (including Compactness and Correctness) that should be evaluated, meaning a good explanation needs to succeed on multiple dimensions simultaneously.

Quantitative evaluation methods exist but are underused: The paper catalogs concrete, measurable evaluation tools that researchers can use to objectively test explanations, yet many papers continue relying on subjective judgments instead of these available quantitative approaches.

Better evaluation could improve AI development: By establishing standardized metrics for explanation quality, developers could optimize AI systems to be both accurate and interpretable during training, rather than treating explainability as an afterthought.