Evaluating Explainability: A Framework for Systematic Assessment and Reporting of Explainable AI Features

Explainability needs systematic evaluation: The paper identifies a critical gap—while AI explainability features are widely used to show how AI makes decisions, there are no standardized methods to evaluate whether these explanations are actually good. The authors propose four specific criteria (consistency, plausibility, fidelity, and usefulness) to fill this gap.

Four key qualities separate reliable explanations from misleading ones: The framework measures whether explanations are consistent (don't wildly change for similar inputs), plausible (match what experts know to be true), faithful (actually reflect how the AI model works internally), and useful (actually help people make better decisions). An explanation could score well on some criteria but fail on others.

Explanations can be wrong even when they seem convincing: Using breast lesion detection as an example, the paper demonstrates that an AI system's explanation of why it flagged a lesion might look reasonable to a doctor but could actually be highlighting the wrong features or not accurately representing the model's internal logic—a critical concern for medical AI where explanations influence clinical decisions.

A "scorecard" approach enables transparency and accountability: The authors propose that explainability methods should come with standardized evaluation scorecards, similar to nutrition labels on food. This would allow hospitals, regulators, and users to compare different AI systems' explanation quality before deployment.

Better evaluation of explanations is essential for trustworthy AI in high-stakes domains: The framework is particularly important for medical AI and other critical applications where people need to understand and verify AI decisions before relying on them in real-world situations.