Explainable AI Methods for Neuroimaging: Systematic Failures of Common Tools, the Need for Domain-Specific Validation, and a Proposal for Safe Application

Common XAI tools fail dangerously in medical imaging: The study found that GradCAM and Layer-wise Relevance Propagation—two of the most widely used explainability methods—systematically failed to correctly identify which brain features the AI model was actually using to make predictions, risking serious misinterpretation in clinical settings.

Prior neuroimaging AI studies may need re-evaluation: Because these flawed explanation methods have been used in previous research on brain MRI analysis, the researchers suggest that conclusions from those studies should be reconsidered, highlighting how validation gaps can propagate through scientific literature.

One-size-fits-all AI explanations don't work across domains: Methods designed for natural images (like photos) don't automatically work for medical images like brain scans. The paper demonstrates that AI explainability tools require domain-specific adaptation and testing rather than direct application across different fields.

Simpler methods can be more trustworthy: Counterintuitively, SmoothGrad—a simpler, gradient-based explanation method that makes fewer assumptions about data structure—proved more reliable than more complex alternatives, suggesting that conceptual simplicity may enhance robustness in high-stakes applications.

Validation frameworks are essential before clinical deployment: The researchers developed a novel testing approach using known signal sources to verify XAI accuracy. This work underscores that explainability methods must be rigorously validated in their specific domain before being trusted for medical decision-making.