Do Explanations Reflect Decisions? A Machine-centric Strategy to Quantify the Performance of Explainability Algorithms

Most explainability methods may not identify the factors that actually drive AI decisions: The paper found that LIME, one of the most popular explanation tools, identified critical regions that had "the lowest impact" on the neural network's actual decision-making, suggesting that visual explanations can be misleading about what the AI is really using to make its choices.

We need better ways to evaluate if explanations are actually truthful: The authors highlight that the field of explainable AI has relied too heavily on "subjective visual interpretation" rather than rigorous quantitative testing, meaning we often can't verify whether an explanation genuinely reflects how an AI system reached its decision.

The paper introduces measurable standards for explanation quality: Two new metrics—Impact Score (measuring whether identified factors actually change the AI's decision) and Impact Coverage (measuring how comprehensively explanations account for what influences the AI)—provide concrete ways to test whether explanations are reliable, moving beyond subjective judgment.

Different explanation methods perform very differently: Testing multiple state-of-the-art methods (LIME, SHAP, Expected Gradients, GSInquire) on the same AI system revealed significant performance variations, suggesting that choosing which explanation tool to use matters substantially for getting accurate insights into AI decision-making.

This work addresses a critical gap in responsible AI: By developing methods to verify that explanations actually match decisions, this research helps ensure that when organizations use XAI tools to build trust or ensure accountability, they're not relying on explanations that might be fundamentally misleading.