Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?

Most popular explanation methods don't reliably help users predict model behavior: The study tested five widely-used explanation approaches (LIME, Anchor, Decision Boundary, Prototype, and Composite methods) and found that "clear evidence of method effectiveness is found in very few cases," suggesting that many tools currently used to explain AI systems may not be as helpful as assumed.

LIME works for tabular data but the picture is mixed overall: Only LIME showed consistent improvement in helping users simulate model behavior on tabular (spreadsheet-style) data, while the Prototype method was effective specifically for counterfactual tests (predicting what would change if inputs were different). This suggests explanation effectiveness depends heavily on both the method and the type of data.

User satisfaction ratings don't predict actual usefulness: The researchers found that subjective ratings—how much users felt the explanations were helpful—did not correlate with whether those explanations actually improved their ability to predict model behavior, highlighting a gap between perceived and real utility.

We're measuring explanation quality the wrong way: The paper emphasizes that "we need to be careful about the metrics we use to evaluate explanation methods," suggesting that current industry standards for assessing explainability tools may be misleading and don't capture what actually matters for responsible AI deployment.

Current explanation methods need significant improvement: The authors conclude there is "significant room for improvement in current methods," indicating that organizations relying on today's explanation tools to understand AI systems should be cautious about their limitations.