Explainable AI (XAI) for Arrhythmia detection from electrocardiograms

Clinical adoption of AI requires explanations doctors can trust: The study found that medical professionals strongly preferred saliency map-based explanations over other formats because they aligned with how doctors already interpret ECGs, highlighting the importance of designing AI explanations around existing clinical workflows rather than imposing new visualization methods.

Not all explanation methods are equally useful in practice: When researchers compared four different explanation techniques (permutation importance, KernelSHAP, gradient-based methods, and DeepLIFT), only the gradient-based and DeepLIFT approaches produced clear, clinically meaningful visualizations—demonstrating that the choice of explanation method significantly impacts whether doctors can actually use and trust the AI system.

High accuracy alone doesn't guarantee real-world performance: The model achieved 98.3% accuracy on one dataset but showed "performance degradation" when tested on a combined dataset from different sources, revealing that AI systems trained in controlled settings may not work reliably across different hospitals or patient populations—a critical concern for clinical deployment.

Explanation consistency matters for trustworthiness: The gradient-based and DeepLIFT methods showed "variability across samples," meaning the AI didn't always highlight the same waveform regions when identifying similar arrhythmias, which could undermine clinician confidence in the system's reliability.

Understanding user needs is essential before building AI systems: The researchers conducted a user needs assessment with medical professionals before selecting explanation methods, demonstrating that responsible AI development requires directly consulting the people who will actually use these systems.