Opening the black box of deep learning: Validating the statistical association between explainable artificial intelligence (XAI) and clinical domain knowledge in fundus image-based glaucoma diagnosis

XAI techniques can validate that AI models focus on clinically relevant features: The study found that deep learning models consistently highlighted anatomically important structures (optic cup, optic disk, blood vessels) in their decision-making process at statistically significantly higher rates than these structures appear in the full image, suggesting the models are learning medically meaningful patterns rather than spurious correlations.

Explainability methods help bridge the trust gap for clinical adoption: By using Class Activation Map (CAM) techniques to visualize where models focus their attention, the researchers demonstrated a pathway to "opening the black box" of deep learning—addressing a key barrier to implementing AI in real healthcare settings where clinicians need to understand and trust diagnostic recommendations.

Multiple explanation techniques and models should be validated together: The study tested five different CAM methods across four different neural network architectures, finding consistent results across all combinations. This suggests that robust validation requires checking multiple explanation approaches rather than relying on a single technique, reducing the risk of false confidence in model interpretability.

Statistical validation of explainability is feasible and important: Rather than relying on subjective visual inspection, the researchers used paired t-tests and correlation analyses to quantitatively verify that model focus areas aligned with clinical knowledge, providing a reproducible framework for validating whether AI explanations are genuinely meaningful.