Explainable AI as a Double-Edged Sword in Dermatology: The Impact on Clinicians versus The Public

Explainable AI can increase over-reliance on AI systems, particularly among non-experts: The study found that lay people showed significantly higher "automation bias" when given LLM explanations—they were more likely to trust the AI when it was correct but also more likely to accept incorrect AI diagnoses. This suggests that detailed explanations may inadvertently make people trust AI too much rather than think critically about its recommendations.

Medical expertise provides protection against AI-induced bias: Experienced primary care physicians remained resilient to automation bias regardless of whether AI explanations were provided or whether the AI was accurate. This indicates that professional training and domain knowledge help clinicians maintain appropriate skepticism toward AI assistance, unlike the general public.

The order in which AI suggestions are presented matters significantly: When AI recommendations were shown first (before the human made their own assessment), both lay users and physicians performed worse when the AI was incorrect. This timing effect suggests that seeing AI suggestions upfront can anchor human judgment in problematic ways.

AI systems balanced for fairness across skin tones can improve overall diagnostic accuracy and reduce health disparities: The fairness-based AI model used in the study improved accuracy and reduced diagnostic disparities, demonstrating that responsible AI design can simultaneously benefit both performance and equity in medical applications.

One-size-fits-all XAI approaches are insufficient for healthcare: The divergent effects of LLM explanations between lay users and physicians highlight that explainability features need to be tailored to the expertise level and role of the end user to be truly beneficial.