Evaluation of Popular XAI Applied to Clinical Prediction Models: Can They be Trusted?

Popular XAI methods may not reliably explain clinical predictions: The study found that two widely-used explainable AI methods showed limited ability to generate explanations that matched actual clinical deterioration triggers recorded in hospital data, raising concerns about whether clinicians can trust these explanations to guide patient care decisions.

XAI explanations need to be clinically meaningful, not just technically accurate: The research emphasizes that for clinical adoption, XAI methods must produce explanations that are "domain-appropriate" and coherent with clinical reality—meaning they should align with how medical professionals understand disease progression, not just match statistical patterns in data.

Consistency of explanations matters for clinical trust: The paper benchmarks XAI methods on consistency, suggesting that if an AI system gives different explanations for similar patients or situations, clinicians cannot reliably use it to support treatment decisions, which is essential for building trust in clinical environments.

Current XAI methods may disrupt rather than support clinical workflows: The study evaluates whether these explanation methods actually fit into how hospitals operate, highlighting that even technically sound explanations are unhelpful if they don't integrate smoothly into existing clinical decision-making processes.

More rigorous evaluation standards are needed before clinical deployment: By conducting the "first benchmarking" of XAI methods against real clinical outcomes from Australian hospital data, this research demonstrates that XAI solutions require validation against actual clinical events—not just theoretical measures—before they should be trusted in patient care settings.