Beyond Satisfaction: From Placebic to Actionable Explanations For Enhanced Understandability

User satisfaction surveys are unreliable measures of explanation quality: The study found that participants rated placebic (vacuous) explanations and actionable (meaningful) explanations as equally satisfying, despite actionable explanations significantly improving objective performance. This reveals a critical gap between how satisfied users feel and whether explanations actually help them understand a system.

Explanations can create a false sense of understanding: Placebic explanations—which sound reasonable but don't provide genuine insight—made users feel informed without improving their mental models or decision-making. This "placebo effect" in AI explanations poses a risk to trustworthiness, as users may have unwarranted confidence in systems they don't truly understand.

Objective performance metrics are essential for evaluating AI transparency: When researchers measured actual task performance and mental model development (not just user surveys), actionable explanations dramatically outperformed placebic ones. This demonstrates that responsible AI evaluation requires measuring what users can actually do with explanations, not just their subjective impressions.

Current XAI evaluation practices may be misleading stakeholders: Many AI systems are evaluated primarily through user satisfaction surveys, which this research suggests are insufficient. Organizations deploying explainable AI systems may be getting false confidence that their transparency efforts are working when users are actually building incomplete or incorrect understandings.

Future AI accountability should combine subjective and objective measures: The authors recommend that evaluations of AI explanations integrate both user satisfaction surveys and objective task performance metrics to provide a complete picture of whether explanations genuinely support human understanding and decision-making.