Order in the Court: Explainable AI Methods Prone to Disagreement

Explainable AI methods frequently disagree with each other: When researchers compared six different explanation methods (LIME, Integrated Gradients, DeepLIFT, Grad-SHAP, Deep-SHAP, and attention-based explanations) on language tasks, they found that "in most cases, none of our chosen methods agree," suggesting that no single method reliably identifies which features are most important to a model's decisions.

Comparing explanations by ranking correlation is unreliable: The paper challenges a common practice of validating explanation methods by checking if they rank features similarly to attention weights. The authors conclude that "rank correlation does not measure the quality of feature-additive methods," meaning this popular validation approach may give false confidence in explanations that aren't actually trustworthy.

Current validation practices may be misleading practitioners: Because different explanation methods produce conflicting results, practitioners relying on any single method without rigorous validation could be making decisions based on explanations that don't accurately reflect how the AI system actually works.

The field needs better diagnostic standards: Rather than comparing methods to each other, the authors recommend that "practitioners should instead use the numerous and rigorous diagnostic methods proposed by the community," emphasizing that more systematic testing approaches are needed to verify whether explanations are genuinely faithful to model behavior.