Explainable AI applications in the Medical Domain: a systematic review

Most XAI solutions use model-agnostic techniques rather than methods built into specific AI models, meaning explanations are added after the fact to help doctors understand AI recommendations—this approach is flexible but may not capture how the AI actually makes decisions.

Deep learning models dominate medical AI applications, yet the paper emphasizes that very few of these AI systems move from research studies into actual clinical practice, suggesting a significant gap between what works in labs and what doctors can realistically use.

Physicians are rarely involved in developing XAI systems: While explainability was designed to build trust with doctors, the review found that few studies included actual physician participation in the design process, meaning these tools may not address what doctors actually need to understand.

Visual and interactive explanations are more effective than text-based ones for helping medical professionals understand AI recommendations, highlighting the importance of user interface design in making AI trustworthy and usable in clinical settings.

Collaboration between medical and AI experts is critical but lacking: The authors call for more interdisciplinary teamwork to create XAI frameworks specifically suited to medicine, rather than applying generic AI explanation techniques to medical problems.