UbiQVision: Quantifying Uncertainty in XAI for Image Recognition

Popular explanation methods like SHAP can be unreliable: The paper demonstrates that SHAP, a widely-used tool for explaining AI decisions in medical imaging, can produce unstable and inconsistent explanations—a critical finding for healthcare applications where doctors rely on these explanations to understand why an AI system made a particular diagnosis.

Uncertainty in explanations varies by medical domain: The research tested their framework across three different medical imaging types (pathology, ophthalmology, and radiology) and found that the reliability of AI explanations differs significantly depending on image quality, resolution, and the specific medical specialty—suggesting that one-size-fits-all approaches to AI transparency in healthcare are insufficient.

New methods can measure how trustworthy explanations actually are: The paper proposes using Dirichlet posterior sampling and Dempster-Shafer theory to quantify uncertainty in explanations, providing a way to flag when an AI system's reasoning is less reliable—helping medical professionals know when to trust an explanation versus when to seek additional verification.

Complex AI models create a transparency trade-off: While sophisticated deep learning architectures (like Vision Transformers) improve diagnostic accuracy, they simultaneously make it harder to understand how the AI reached its conclusions, highlighting a fundamental challenge in deploying advanced AI responsibly in high-stakes medical settings.