A Meta-Analysis of the Utility of Explainable Artificial Intelligence in Human-AI Decision-Making

Explainable AI shows modest but measurable benefits: The meta-analysis found a statistically positive impact of XAI on users' decision-making performance overall, suggesting that efforts to make AI systems more transparent do help people make better decisions when working with AI.

Explanations alone may not be enough: Surprisingly, the study found "no effect of explanations on users' performance compared to sole AI predictions," indicating that simply adding explanations to AI recommendations doesn't automatically improve outcomes—the quality and relevance of those explanations matters significantly.

Task type influences effectiveness: The research identified that human-AI decision-making tends to perform better with text-based data compared to other data types, suggesting that XAI techniques may work better for certain applications (like document review or content moderation) than others.

Inconsistent results across studies highlight a gap: The meta-analysis was motivated by the observation that different XAI studies report widely varying results—from significant improvements to negligible effects—indicating that the field still lacks clear understanding of when and why explanations actually help users.

More research needed on underlying mechanisms: The authors call for future work to investigate why XAI sometimes helps and sometimes doesn't, emphasizing that developing truly effective AI explanations requires deeper understanding of what makes explanations meaningful to human decision-makers.