User Trust on an Explainable AI-based Medical Diagnosis Support System

Explainability alone doesn't guarantee trust: Despite providing causal explanations for AI diagnoses, radiologists only agreed with the model 46.4% of the time and gave a self-reported trust score of just 3.2 out of 5.0, suggesting that showing why an AI made a decision is necessary but insufficient for building clinician confidence.

There's a significant gap between AI performance and clinical acceptance: The system achieved 74.1% accuracy, but experts disagreed with it more often than they agreed with it, highlighting that technical accuracy metrics don't translate directly into whether medical professionals will actually rely on AI systems in practice.

Interactive correction mechanisms may be key to improvement: The study design allowed radiologists to correct explanations they disagreed with, suggesting that two-way dialogue between humans and AI systems—rather than one-directional explanations—could be an important pathway to building appropriate trust in medical AI.

Trust measurement matters for responsible deployment: The researchers measured trust through multiple methods (agreement rates, expert feedback on explanations, and self-reported scores), demonstrating that responsible AI evaluation in healthcare requires going beyond accuracy metrics to understand whether real users will actually use and rely on these systems appropriately.