ExplainabilityTransparencyTrustworthinesscs.HC
A User-Centric Analysis of Explainability in AI-Based Medical Image Diagnosis
Julia Wagner, Tim Schlippe
March 31, 2026
arXiv.org
Key Takeaways
-
Doctors overwhelmingly want AI explanations: 88% of the 33 physicians surveyed agreed that AI systems should explain their diagnoses, with 64% strongly agreeing, indicating that explainability is not a nice-to-have feature but a fundamental requirement for clinical adoption.
-
Simple combinations work best: A combination of bounding boxes (visual highlighting) and written reports outperformed other explainability methods across multiple measures including understandability, completeness, speed, and practical applicability—suggesting that complex AI explanations don't always serve users better.
-
Explainability alone cannot prevent misplaced trust in errors: Despite having access to various explanation methods, 50% of physicians still trusted false AI diagnoses, revealing a critical gap where even well-explained AI systems can mislead medical professionals and highlighting the need for additional safeguards beyond explanation.
-
Current AI systems lack adequate explanation capabilities: The paper notes that state-of-the-art AI systems in medicine often lack "optimal explanation and visualization," meaning that even top-performing AI tools are being held back from clinical use not by accuracy but by their inability to communicate their reasoning to doctors.
Abstract
In recent years, AI systems in the medical domain have advanced significantly. However, despite outperforming humans, they are rarely used in practice since it is often not clear how they make their decisions. Optimal explanation and visualization of the decision process are often lacking. Therefore, we conducted a comparative user-centric analysis of the latest state-of-the-art textual, visual and multimodal explainable artificial intelligence (XAI) methods for medical image diagnosis. Our survey of 33 physicians showed that 88% agree that it is important that AI explains the diagnosis -- 64% even strongly agree. A combination of bounding box and report is rated better than the other tested XAI methods in the evaluated aspects understandability, completeness, speed, and applicability. We even tested the potential negative impact of false AI-based medical image diagnoses and found that 50% of the participants trusted false AI diagnoses over all tested XAI methods.