On the Design and Evaluation of Human-centered Explainable AI Systems: A Systematic Review and Taxonomy

One-size-fits-all explanations don't work: The paper emphasizes that XAI systems must be designed differently for users with varying levels of AI expertise—what works for a data expert may confuse an AI novice. This suggests companies deploying AI systems need to tailor explanations to their specific user audiences rather than providing generic technical descriptions.

Technical performance alone is insufficient: Current AI evaluation processes focus heavily on whether systems work accurately, but this review of 65 user studies found that human-centered evaluation metrics—including how users feel about the system, whether they understand it, and whether it's actually usable—are equally critical and often overlooked in practice.

Explanations are separate from the AI system itself: A key finding distinguishes between the core AI system and the explanation layer added on top of it. This matters because a highly accurate AI system can still fail users if its explanations are poor, meaning organizations need to invest in explanation quality as a distinct design priority.

Multiple dimensions of "understanding" matter: The paper identifies five distinct evaluation categories (user affect, cognition, usability, interpretability, and explanation quality) rather than treating "explainability" as a single concept. This suggests responsible AI requires assessing whether users actually trust, comprehend, and can effectively use AI systems—not just whether explanations exist.

User studies should guide XAI design: The systematic review of real user studies provides practical guidance that developers should follow when building explainable systems, rather than relying solely on technical benchmarks or assumptions about what users need.