Evaluating the Utility of Model Explanations for Model Development

Explanations don't automatically improve decision-making: Despite being a major goal of explainable AI, the study found no significant improvement in users' ability to select better models or simulate counterfactuals when given saliency map explanations—even when using an "oracle" explanation designed to be maximally clear and helpful.

Current evaluation methods may be misleading: The paper highlights that most explainability research relies on subjective measures of how useful people think explanations are, rather than testing whether explanations actually help with real tasks. This gap between perception and performance is a critical blind spot in the field.

Explanations do help with model understanding, just not decision-making: While saliency maps didn't improve task performance, they did help users more accurately describe how models work, suggesting explanations have value for transparency and comprehension even if they don't translate to better practical decisions.

Risk of misunderstanding with visual explanations: The finding that even well-designed saliency maps didn't improve performance raises concerns that users might misinterpret these visualizations or place unwarranted confidence in them, potentially creating a false sense of understanding.

Responsible AI deployment requires caution: Before widely adopting explainability tools in high-stakes settings, organizations should rigorously test whether explanations actually improve human decision-making in their specific contexts, rather than assuming they will.