The Impact of Imperfect XAI on Human-AI Decision-Making

Incorrect explanations can mislead even when AI advice is correct: The study found that explanations themselves can be wrong independently of whether the AI's recommendation is accurate—a gap in XAI research that previous studies largely overlooked. This means users can be deceived into trusting faulty reasoning, even if the AI happens to reach the right conclusion.

Expertise level significantly affects how people respond to bad explanations: The research with 136 participants showed that a decision-maker's level of expertise influences their reliance on AI and overall team performance when exposed to imperfect explanations, suggesting that XAI design cannot be one-size-fits-all.

Assertive explanations pose particular risks: The study examined how the confidence level or assertiveness of explanations impacts human decision-making, revealing that overly confident or assertive explanations may increase the potential for deception during human-AI collaboration.

Imperfect XAI undermines appropriate reliance: Rather than helping humans calibrate their trust in AI systems, incorrect explanations can distort decision-makers' judgment about when to rely on AI recommendations, ultimately harming team performance in collaborative settings.

Current XAI design guidelines may be insufficient: The findings highlight that designing human-centered AI tools requires acknowledging and addressing the risk that explanations themselves can be flawed, not just focusing on cases where AI advice is wrong.