A Survey on Human-Centered Evaluation of Explainable AI Methods in Clinical Decision Support Systems

Current XAI explanations improve trust but create practical problems: While over 80% of studies found that AI explanations increased clinician trust and diagnostic confidence, the same explanations frequently increased cognitive load and didn't align with how doctors actually reason about cases, suggesting that current approaches solve one problem while creating others.

Most XAI research uses small, limited studies: The survey found that clinician sample sizes typically remained below 25 participants, raising questions about whether findings from these small-scale studies generalize to real clinical settings where decisions affect many patients.

Post-hoc explanations dominate but may not be ideal: Over 80% of evaluated XAI methods use post-hoc, model-agnostic approaches (like SHAP and Grad-CAM), which are added after AI systems are built rather than designed into them from the start—this may explain why explanations often misalign with clinical reasoning.

Evaluation methods are inconsistent and incomplete: The survey reveals that existing XAI evaluations are "inconsistently evaluated," meaning there's no standard way researchers assess whether explanations actually help clinicians make better decisions in practice, making it difficult to compare different approaches.

Future XAI needs input from all stakeholders, not just technologists: The authors propose a new evaluation framework that incorporates perspectives from clinicians, patients, and other stakeholders alongside human-computer interaction principles, suggesting that better XAI requires collaboration beyond just AI developers.