The Blame Problem in Evaluating Local Explanations, and How to Tackle it

The field of AI explainability lacks rigorous evaluation standards. The paper identifies that because there are no optimal measures for evaluating explanation techniques, it's easy for researchers to propose new methods without proving they actually work better than existing ones—making it difficult to determine real progress in the field.

Most popular evaluation methods suffer from the "blame problem." The paper reveals that robustness tests, synthetic dataset evaluations, model randomization tests, and human studies all struggle with a fundamental flaw: they cannot definitively prove that an explanation technique is actually capturing what makes a model make its decisions, rather than just appearing to do so.

Evaluations based on interpretable models are more trustworthy, but still imperfect. The only evaluation category that avoids the "blame problem" involves testing explanations against models whose decision-making is already fully understood (like simple decision trees). However, the paper acknowledges this approach has its own significant limitations.

Evaluating AI explanations remains an unsolved challenge. The paper concludes that determining whether an explanation technique truly works is still an open research problem, suggesting that current claims about explanation quality should be viewed with appropriate skepticism until better evaluation methods are developed.