ExplainabilityTrustworthinessAccountabilitycs.HC
Usability Testing of an Explainable AI-enhanced Tool for Clinical Decision Support: Insights from the Reflexive Thematic Analysis
Mohammad Golam Kibria, Lauren Kucirka, Alison Stuebe, Javed Mostafa
April 7, 2025
IEEE International Conference on Healthcare Informatics
Key Takeaways
-
Explainability is a critical barrier to AI adoption in healthcare: The study found that unclear AI explanations significantly discourage clinicians from accepting AI-enhanced clinical decision support tools, suggesting that technical performance alone is insufficient for real-world implementation.
-
One-size-fits-all explanations don't work for clinicians: Through interviews with 20 U.S.-based clinicians, the research identified that different clinical users have varying preferences for how AI decisions should be explained, indicating that customization of explainability features is necessary for effective tool design.
-
A practical framework for explainability is needed: The study developed a concrete framework and operational definition of explainability specifically tailored to clinical settings, providing actionable guidance for developers on what "good" AI explanations should look like in healthcare contexts.
-
Usability testing with actual end-users reveals implementation gaps: By conducting qualitative research directly with clinicians rather than relying solely on technical metrics, the study uncovered that real-world adoption challenges differ from what developers might anticipate, emphasizing the importance of involving healthcare professionals in AI tool design.
Abstract
Artificial intelligence-augmented technology represents a considerable opportunity for improving healthcare delivery. Significant progress has been made to demonstrate the value of complex models to enhance clinicians` efficiency in decision-making. However, the clinical adoption of such models is scarce due to multifaceted implementation issues, with the explainability of AI models being among them. One of the substantially documented areas of concern is the unclear AI explainability that negatively influences clinicians` considerations for accepting the complex model. With a usability study engaging 20 U.S.-based clinicians and following the qualitative reflexive thematic analysis, this study develops and presents a concrete framework and an operational definition of explainability. The framework can inform the required customizations and feature developments in AI tools to support clinicians` preferences and enhance their acceptance.