A Meta Survey of Quality Evaluation Criteria in Explanation Methods

There is no agreed-upon standard for evaluating AI explanations: The paper identifies a "critical shortage of rigour in the field" because existing evaluation criteria focus on single explanation methods rather than enabling meaningful comparisons between different approaches, making it difficult to determine which explanation method is actually better.

The most accurate AI models are often the hardest to understand: The paper notes that "the most accurate AI models are opaque with low transparency and comprehensibility," creating a fundamental tension in AI development where improved performance comes at the cost of explainability—making quality explanations essential for detecting bias and managing uncertainty.

Explanations should be evaluated across three distinct dimensions: The research proposes grouping evaluation criteria into three quality aspects—model (how well the explanation reflects the AI's actual decision-making), explanation (the quality of the explanation itself), and user (whether humans can actually understand and trust it)—rather than treating explanation quality as a single concept.

"Appropriate trust" should be the ultimate measure of explanation quality: The paper suggests using appropriate trust as a meta-criterion to evaluate subjective evaluation criteria, meaning a good explanation is one that helps users develop the right level of confidence in the AI system—neither blind trust nor unwarranted skepticism.

Four core evaluation criteria are widely accepted across the field: Despite the lack of consensus, the research identifies four commonly accepted criteria groups that cover all aspects of explanation quality, suggesting some foundation exists for standardization.