On the stability, correctness and plausibility of visual explanation methods based on feature importance

Existing explanation evaluation metrics often contradict each other: The paper reveals that current methods for assessing whether AI explanations are "stable" (consistent), "correct" (accurate), and "plausible" (believable) don't always agree with one another, suggesting there's no universal standard for what makes a good explanation.

Current metrics may miss important local behavior: The researchers found that popular evaluation metrics fail to capture how AI models behave in specific local regions around individual predictions, which is critical for understanding whether explanations are truly reliable for particular decisions.

Stability and correctness metrics have significant limitations: The paper demonstrates that some widely-used metrics for measuring whether explanations remain consistent and accurate have fundamental flaws, and proposes improved alternatives that better reflect real-world model behavior.

The "explanation quality" problem remains unsolved: Rather than settling on one best approach, this work highlights a fundamental challenge in responsible AI: we still lack consensus on how to rigorously evaluate whether the explanations AI systems provide are actually trustworthy and meaningful.

Implications for AI accountability: These findings suggest that organizations relying on explanation methods to justify AI decisions should be cautious about which evaluation metrics they trust, as different metrics may give conflicting signals about whether an explanation is genuinely reliable.