XAI Evaluation Cards: A Practical Method for Designing Human-Centred XAI Evaluations

Evaluation of XAI systems has been fragmented and inconsistent: The paper analyzed 82 existing studies and found that researchers were selecting evaluation dimensions in an "ad hoc and fragmented manner," suggesting there was no standardized approach to assessing whether explainable AI actually works for people.

A practical framework can help teams evaluate XAI more systematically: The authors developed 36 evaluation cards organized around key dimensions that researchers and designers can use to systematically plan how to test XAI systems, rather than making arbitrary choices about what to measure.

Multidisciplinary input improves XAI evaluation: Testing the card-sorting method with 13 researchers across five projects showed that involving people from different fields (HCI, computer science, design, and social science) leads to more comprehensive assessments of whether XAI systems actually help users understand AI decisions.

Practical tools can bridge the gap between theory and practice: By creating printable cards and an online repository of evaluation methods from previous studies, the authors made it easier for teams developing AI systems to apply rigorous, human-centered evaluation approaches without starting from scratch.

There is no one-size-fits-all evaluation approach: While the framework provides structure, the paper emphasizes that teams should prioritize different evaluation aspects based on their specific context, suggesting that responsible AI evaluation requires thoughtful customization rather than checklist compliance.