Transparency and Trust in Human-AI-Interaction: The Role of Model-Agnostic Explanations in Computer Vision-Based Decision Support

AI systems can make correct predictions for the wrong reasons: The paper found that malaria detection models sometimes identified diseases correctly while relying on "questionable or irrelevant data features" in images, highlighting that accuracy alone doesn't guarantee trustworthy decision-making in high-stakes domains like medical diagnostics.

Different AI models explain their decisions differently: The research discovered "significant discrepancies in how different deep learning models explain the same prediction," meaning two models could reach identical conclusions through entirely different reasoning—a critical concern when deploying AI in healthcare where understanding the basis for decisions matters.

Explainability tools can reveal hidden problems: By using Explainable AI (XAI) techniques to "open the black box," researchers uncovered flaws in how models were making decisions that wouldn't have been apparent from performance metrics alone, demonstrating that transparency mechanisms serve as quality control tools.

Transparency builds justified trust in AI systems: The paper's theoretical discussion emphasizes that XAI can support trust in AI systems "especially through an increase" in transparency, suggesting that trust should be earned through understanding rather than blind acceptance of algorithmic outputs.

XAI is particularly important for medical AI applications: The focus on malaria detection from blood smear images underscores that explainability becomes essential when AI systems influence patient care decisions, where clinicians need to understand not just what the AI recommends, but why.