Beware of "Explanations" of AI

Explanations alone don't guarantee trust or safety. While explainable AI (XAI) is often promoted as a solution for transparency and responsible AI adoption, poorly designed explanations can actually cause harm—including wrong decisions, privacy violations, and manipulation—rather than fostering genuine understanding.

"Good" explanations depend heavily on context and stakeholders. There is no universal standard for what makes an explanation effective; the quality depends on specific goals, who is using it, and the situation. What works for a regulator may not work for an end user, requiring tailored approaches rather than one-size-fits-all solutions.

Many AI explanations suffer from serious quality problems. Explanations can be unfaithful (not accurately representing how the AI actually works), irrelevant (addressing the wrong question), or incoherent (confusing rather than clarifying), undermining their intended purpose.

Psychological alignment matters but is difficult to achieve in practice. The concept of "mental model alignment"—ensuring explanations match how people actually think—offers guidance, but social and technical factors make this challenging to implement successfully in real-world settings.

Stakeholders should approach AI explanations with caution. Rather than assuming explanations automatically improve responsible AI adoption, organizations should carefully evaluate explanation quality and recognize that poorly implemented explanations can exacerbate existing harms.