Critical Empirical Study on Black-box Explanations in AI

Post-hoc explanations may provide incomplete or misleading information: The paper's empirical testing found that explanations added after-the-fact to black-box AI models tend to give "partial and biased information" about how the algorithm actually works, rather than revealing its true underlying mechanisms.

Explanations can be manipulated to mislead users: The research demonstrates that post-hoc explanations are vulnerable to manipulation and information withholding, with the potential to "divert users' attention" away from how decisions are actually being made—raising concerns about whether these explanations genuinely serve the public interest.

What people say they understand differs from what they actually understand: The study emphasizes the importance of measuring "behavioral indicators" (what people actually do) alongside "self-reported perceived indicators" (what people say they understand), suggesting that current methods of testing whether explanations work may be giving a false sense of success.

The debate between transparent-by-design vs. explained-after-the-fact matters for society: The paper highlights a fundamental tension in AI development: whether it's better to build inherently interpretable AI systems from the ground up, or to rely on explanations bolted onto complex black-box models—a choice that will significantly influence how AI systems are built and deployed in the future.