Investigating the fidelity of explainable artificial intelligence methods for applications of convolutional neural networks in geoscience

Popular explainability methods have significant blind spots: The study found that common XAI methods suffer from overlooked theoretical limitations—including "gradient shattering," inability to distinguish whether inputs positively or negatively influence predictions, and failure to account for zero inputs—which could lead researchers to misunderstand how CNNs actually make decisions in geoscience applications.

Ground truth testing is essential for validation: By testing XAI methods against an "idealized attribution benchmark" where the correct explanation was known in advance, the researchers demonstrated that objective performance assessment is possible and necessary before trusting these methods in real-world applications like climate prediction.

XAI methods are not interchangeable: The intercomparison revealed relative strengths and weaknesses across different XAI approaches, suggesting that practitioners cannot assume all popular explainability techniques will perform equally well—method selection matters for the reliability of explanations.

Explainability limitations could distort scientific understanding: When applied to predicting atmospheric rivers in climate simulations, the study showed that flawed XAI explanations could create a "distorted picture" of CNN decision-making, with potential consequences for how scientists interpret and trust AI predictions in high-stakes geoscience research.

Careful consideration of XAI limitations is now a best practice requirement: The authors emphasize that awareness of these theoretical limitations should guide how researchers apply and interpret explainability methods, rather than accepting explanations at face value.