Elucidating Discrepancy in Explanations of Predictive Models Developed using EMR

XAI methods don't always align with clinical expertise: The study found discrepancies between what explainability methods identify as important factors and what clinical experts consider clinically relevant, highlighting that transparency alone doesn't guarantee trustworthiness in healthcare AI systems.

Multiple explanation methods can produce conflicting results: Different state-of-the-art XAI techniques applied to the same clinical prediction models generated different explanations, suggesting that practitioners cannot assume any single explanation method will reliably reveal how a model actually makes decisions.

Clinical context matters for AI validation: The research emphasizes that evaluating AI explanations requires domain expertise—discrepancies between technical explanations and clinical knowledge can stem from both technical issues (like how models process EMR data) and legitimate clinical reasoning that algorithms may not capture.

Transparency is necessary but insufficient for clinical adoption: While explainability methods are important for addressing the "black box" problem in healthcare ML, this study demonstrates that achieving genuine trustworthiness requires agreement between algorithmic explanations, clinical evidence, and expert judgment rather than relying on explanations alone.

Interdisciplinary collaboration is essential for responsible clinical AI: The paper underscores that developing trustworthy decision support systems requires ongoing dialogue between AI developers, clinicians, and domain experts to identify and resolve cases where algorithms and clinical knowledge diverge.