Explainability's Gain is Optimality's Loss? -- How Explanations Bias Decision-making

Explanations can introduce bias rather than reduce it: The paper's field experiment found that feature-based explanations (which show which factors the AI considered most important) actually led decision-makers to experience confirmation bias—they were more likely to trust predictions that aligned with their existing beliefs, even when those predictions weren't optimal.

Explanations may create disparate impact: The research demonstrates that explanations don't affect all decision-makers equally. Different people's confidence in AI predictions varied based on whether the explanations confirmed their prior assumptions, potentially leading to inconsistent and unfair treatment of similar cases.

Better communication doesn't always mean better decisions: While explanations are often promoted as a way to help humans understand and work better with AI systems, this study shows they can actually lead to "sub-optimal and biased decision outcomes" by triggering confirmation bias rather than encouraging careful evaluation of alternatives.

The semantics of explanations matter: The paper suggests that how explanations are framed—particularly when they're presented as causal relationships between features and outcomes—can "induce leakage from the decision-maker's prior beliefs," meaning people unconsciously filter the information through what they already believe to be true.

Organizations should reconsider how they deploy AI explanations: These findings suggest that simply adding explanations to AI systems without understanding their psychological effects may backfire, making human decision-makers less objective rather than more informed.