Designing for Complementarity: A Conceptual Framework to Go Beyond the Current Paradigm of Using XAI in Healthcare

Explainability alone isn't enough to build trust: The paper identifies that simply making AI systems transparent can lead to either "over-reliance" (trusting AI too much) or "under-reliance" (trusting it too little), suggesting that healthcare organizations need to think beyond just explaining how AI makes decisions.

A three-part explanation approach works better: The framework combines Feature Importance (what factors the AI considered), Counterexample Explanations (cases where the AI would decide differently), and Similar-Case Explanations (comparable real cases), offering clinicians multiple ways to understand AI recommendations rather than a single explanation type.

Clinicians should be partners in AI design, not afterthoughts: The study used participatory design with healthcare professionals to develop the prototype, indicating that involving actual end-users in building AI systems—rather than designing them in isolation—leads to tools that better support clinical decision-making.

The goal is "complementarity," not replacement: Rather than positioning AI as either a tool to blindly follow or to ignore, the framework aims to create systems where AI and clinician expertise work together, with each compensating for the other's limitations in the decision-making process.

Black-box AI creates real adoption barriers in healthcare: The paper emphasizes that opacity in AI systems raises ethical and legal concerns that actively prevent healthcare organizations from adopting these tools, making explainability a practical necessity, not just a nice-to-have feature.