ExplainabilityTransparencyTrustworthinesscs.AI
Evaluating the Explainable AI Method Grad-CAM for Breath Classification on Newborn Time Series Data
Camelia Oprea, Mike Grüne, Mateusz Buglowski, Lena Olivier, Thorsten Orlikowsky, Stefan Kowalewski, Mark Schoberer, André Stollenwerk
May 13, 2024
IFAC-PapersOnLine
6 citations
Key Takeaways
-
Explainability methods don't automatically build trust: The study found that Grad-CAM, a popular explanation technique, had limited perceived usefulness among stakeholders examining newborn ventilation data, suggesting that simply adding explanations to AI systems isn't enough to achieve genuine transparency in medical settings.
-
Different users have different needs: The paper reveals that various stakeholders (likely clinicians, engineers, and others) wanted different types of explanations, with many participants requesting "more in-depth explanations" beyond what Grad-CAM provided, indicating a one-size-fits-all approach to AI explainability is insufficient.
-
Medical AI transparency remains challenging: Even in a focused application like breath classification for newborn care, achieving actual transparency proved difficult, highlighting that the gap between AI decision-making and human comprehension is a persistent problem that current explainability methods have not fully solved.
-
User studies are essential for evaluating AI explanations: Rather than assuming an explanation method works, this research directly tested it with real stakeholders, demonstrating that empirical evaluation of explainability tools—not just technical validation—is necessary before deploying them in healthcare.
Abstract
With the digitalization of health care systems, artificial intelligence becomes more present in medicine. Especially machine learning shows great potential for complex tasks such as time series classification, usually at the cost of transparency and comprehensibility. This leads to a lack of trust by humans and thus hinders its active usage. Explainable artificial intelligence tries to close this gap by providing insight into the decision-making process, the actual usefulness of its different methods is however unclear. This paper proposes a user study based evaluation of the explanation method Grad-CAM with application to a neural network for the classification of breaths in time series neonatal ventilation data. We present the perceived usefulness of the explainability method by different stakeholders, exposing the difficulty to achieve actual transparency and the wish for more in-depth explanations by many of the participants.