A Systematic Review of User-Centred Evaluation of Explainable AI in Healthcare

Most XAI systems in healthcare lack real-world validation: The paper found that despite advances in explainable AI technology, "the practical value of XAI methods remains under-explored and insufficiently validated in real-world settings," meaning many systems haven't been tested with actual users in clinical environments before deployment.

There are no clear standards for evaluating whether explanations actually help users: The review identified a critical gap—there are "no clear guidelines on how to design an evaluation with users"—which means healthcare organizations often don't know how to properly test whether AI explanations are truly useful and trustworthy for doctors, nurses, and other clinicians.

User experience matters as much as technical accuracy: The framework developed from 82 user studies emphasizes that evaluation must focus on "well-defined, atomic properties that characterise the user experience of XAI," highlighting that an explanation can be technically correct but still fail if it doesn't work for the people actually using it in clinical practice.

Different healthcare contexts need different evaluation approaches: The guidelines provide "context-sensitive" strategies, recognizing that evaluating an AI explanation system for radiologists requires different methods than evaluating one for primary care doctors, reflecting that one-size-fits-all approaches to XAI validation are insufficient.

Healthcare teams need practical tools to evaluate XAI properly: The paper provides "actionable guidelines to support interdisciplinary teams in designing and implementing effective evaluation strategies," offering concrete steps that hospitals and healthcare organizations can follow when assessing whether an AI explanation system is actually trustworthy and usable.