A Human-Grounded Evaluation of SHAP for Alert Processing

Explanation methods don't automatically improve decision-making: Despite SHAP being a well-regarded explanation technique in AI research, the study of 159 participants found no statistically significant difference in alert processing performance when SHAP explanations were provided versus when they were not available.

Model confidence scores still dominate human judgment: The research revealed that participants relied primarily on the model's confidence score as their main source of evidence, suggesting that even when explanations are provided, humans may default to simpler signals rather than engaging with detailed explanation information.

There's a gap between theory and practice: The findings highlight an important disconnect—explanation methods that are popular in academic AI communities may not translate into practical benefits for real-world users, indicating the need for more human-centered testing before deployment.

Domain expertise alone isn't sufficient for using explanations: Even participants with basic knowledge of explainable machine learning did not show improved performance with SHAP explanations, suggesting that understanding how an explanation method works doesn't guarantee it will be useful in actual decision-making tasks.

Responsible AI requires empirical validation with real users: This study demonstrates the importance of testing AI interpretability tools with actual domain experts in realistic scenarios rather than assuming that theoretically sound methods will automatically help people make better decisions.