How can I choose an explainer? An Application-grounded Evaluation of Post-hoc Explanations

Popular explanation methods may not improve decision-making in practice: The study found that LIME, SHAP, and TreeInterpreter—three widely-used explainability techniques—did not consistently improve fraud analysts' performance compared to simpler approaches, challenging the assumption that more explanation is always better.

Sometimes less information leads to better decisions: When fraud analysts had access to only transaction data without model scores or explanations, they achieved the highest decision accuracy, suggesting that explanations can sometimes introduce confusion rather than clarity in real-world decision-making tasks.

Explainers should be evaluated by their actual impact, not just technical properties: The paper demonstrates that XAI methods are often chosen based on technical qualities like "fidelity" or "robustness," but these don't necessarily translate to helping humans make better decisions—what matters is real-world performance on actual tasks.

Evaluation methodology matters for responsible AI deployment: The study introduces XAI Test, a practical evaluation framework that tests explanations with real end-users, real data, and deployed models, rather than in laboratory settings—highlighting the importance of testing AI systems in their actual use contexts before deployment.

One-size-fits-all explanations may not work: The research suggests that organizations need to carefully test which explanation methods actually help their specific users and tasks, rather than assuming popular XAI techniques will universally improve human-AI decision-making.