Evaluating Explainable AI on a Multi-Modal Medical Imaging Task: Can Existing Algorithms Fulfill Clinical Requirements?

Existing explainability tools fall short in clinical settings: The study evaluated 16 commonly-used heatmap algorithms and found that none adequately fulfilled clinical requirements for explaining AI decisions on multi-modal medical images, suggesting current explainable AI (XAI) methods may not be ready for direct clinical deployment without significant improvements.

Multi-modal medical imaging presents unique explanation challenges: Medical images often combine multiple imaging types (modalities) that show different aspects of the same condition. Current explanation methods don't properly account for which modality is most important for a given diagnosis, creating a gap between how AI makes decisions and how clinicians need to understand them.

Clinician input is essential for validating AI explanations: The research combined computational testing with an actual clinician user study, demonstrating that technical metrics alone are insufficient—real clinical users must evaluate whether explanations match their professional needs and decision-making processes.

A new evaluation framework is needed: The authors introduced the modality-specific feature importance (MSFI) metric to better assess whether explanation algorithms correctly identify which imaging modality and which specific features within that modality drove the AI's decision, providing a more clinically-grounded standard for evaluation.

Better XAI design can improve clinical trust and safety: By establishing clear clinical requirements and evaluation methods, the research provides a roadmap for developing explanation algorithms that clinicians can actually rely on, potentially enabling safer integration of AI into medical decision-making workflows.