Outlining the Design Space of Explainable Intelligent Systems for Medical Diagnosis

Medical professionals view AI systems as "black boxes" that create trust barriers: The paper identifies that doctors feel concerned about relying on system-generated diagnostic results because they cannot understand how the systems reach their conclusions, highlighting a critical gap between AI capabilities and clinical adoption.

Explainability should mirror how doctors actually think: The research proposes that AI systems will be perceived as more trustworthy and explainable if they are designed to think and act like medical professionals, rather than simply providing technical explanations of their algorithms.

Data prioritization is central to medical diagnosis: The interview study with six medical professionals revealed that doctors don't treat all patient data equally—they actively prioritize among various types of information as a core part of their diagnostic reasoning, suggesting that explainable AI systems should reflect this selective, hierarchical approach to data.

Human-centered design is essential for medical AI: The paper emphasizes that understanding how medical professionals actually work with information is necessary before designing explainable systems, moving beyond purely technical approaches to XAI toward designs grounded in clinical practice.

Current AI explanations may not address clinician needs: The findings suggest that standard AI explanations may not satisfy doctors' actual requirements for understanding diagnostic recommendations, pointing to a need for redesigning how medical AI systems communicate their reasoning.