Position Paper: Integrating Explainability and Uncertainty Estimation in Medical AI

Medical AI systems currently lack transparency about confidence levels: The paper identifies a critical gap where existing AI tools explain what they predict but fail to communicate how confident they are in those predictions—a fundamental mismatch with how doctors actually practice medicine, where uncertainty is central to clinical decision-making.

Two separate AI approaches need to work together: Explainability techniques (which help doctors understand AI reasoning) and uncertainty estimation (which measure prediction reliability) have developed independently. The paper argues these must be integrated into "Explainable Uncertainty Estimation" (XUE) to build genuine trust in medical AI systems.

Better communication of AI confidence could improve adoption: By presenting uncertainty in "clinically meaningful" ways—aligned with how physicians already think about risk and confidence—medical AI systems could become more trustworthy and actually usable in real hospital settings, rather than remaining experimental tools.

Multiple technical improvements are needed: The paper identifies specific directions including better handling of different data types (images, text, numbers), clearer visualization methods that work across different AI models, and decision support systems that explicitly account for uncertainty when recommending actions.

Responsible medical AI requires alignment with clinical reality: The core message is that trustworthy medical AI cannot simply maximize prediction accuracy; it must acknowledge and communicate the inherent uncertainties in medicine itself, reflecting the complexity doctors face daily.