Quality Without Usefulness: LLM-Generated XAI Narratives as Trust Heuristics Rather Than Decision Aids

High-quality explanations don't improve decision-making: Despite LLM-generated explanations scoring well on quality metrics like coherence and comprehensibility, they failed to improve task accuracy across all five experiments (2,730 judgments), suggesting that readable explanations alone don't translate to better human decisions.

Explanations create false confidence without substance: The study found that LLM-generated narratives inflated users' self-reported confidence in their decisions, even when accuracy didn't improve. A control experiment revealed this confidence boost came simply from the presence of text rather than its actual content, indicating a "placebo effect."

Explanations can mask model failures: In out-of-distribution detection tasks, natural language explanations actually reduced people's ability to identify when the AI system was making unreliable predictions, providing "false reassurance that masks model failure"—a potentially dangerous outcome for real-world applications.

Current XAI evaluation metrics are insufficient: The paper demonstrates a "Quality-Usefulness Gap," showing that standard text-quality metrics (plausibility, coherence, comprehensibility) don't capture whether explanations actually help people make better decisions or use AI systems more responsibly.

Evaluation must include real-world task performance: The authors argue that assessing explainable AI systems requires moving beyond measuring explanation quality alone to measuring downstream impacts—whether explanations actually improve human decision-making and help people appropriately trust or distrust AI predictions.