Looking deeper into interpretable deep learning in neuroimaging: a comprehensive survey

Deep learning models in neuroimaging show improved performance over traditional methods, but lack transparency that is critical for healthcare applications. The paper emphasizes that while these models learn directly from raw brain imaging data, their "black box" nature creates safety concerns for real-world clinical deployment.

Explainable AI (XAI) methods help reveal how models make decisions, but researchers still don't fully understand what these explanations actually show. The survey identifies a key gap: there is uncertainty about "what aspect of model learning a post hoc method reveals and how to validate its reliability," meaning we cannot always trust that interpretability tools are showing us what we think they are.

Interpretability methods have enabled neuroimaging researchers to identify brain changes relevant to model predictions. The paper notes that recent studies have successfully used interpretability techniques to capture "anatomical and functional brain alterations most relevant to model predictions," demonstrating practical value for understanding disease mechanisms.

Current interpretability practices have significant limitations that need addressing. The survey identifies gaps between what interpretability methods promise and what they actually deliver, suggesting that current approaches may not be sufficient for responsible deployment in healthcare without further validation and guidance.