Black Box Model Explanations and the Human Interpretability Expectations -- An Analysis in the Context of Homicide Prediction

XAI methods don't always align with expert judgment: When six different explainability methods were applied to a homicide prediction model, they achieved only approximately 48% agreement with human experts' expectations, suggesting that automated explanations may miss important contextual factors that domain specialists consider crucial.

Most expert expectations were partially met despite disagreement: Despite the relatively low agreement rate between XAI methods and experts, 75% of human experts' interpretability expectations were met overall, indicating that while explanations may differ in specifics, they generally point toward reasonable conclusions about model behavior.

Different XAI methods produce different explanations: The study compared six popular XAI tools (SHAP, LIME, ELI5, DALEX, CIU, and Skater) and found they generate varying forms of explanations and feature rankings, meaning the choice of explanation method matters significantly for how stakeholders understand AI decisions.

Context matters for interpreting AI explanations: The research emphasizes that explanations must be grounded in problem context—in this case, homicide prediction—suggesting that generic XAI methods need domain expertise to be properly understood and validated by decision-makers.

Explainability alone doesn't guarantee trustworthiness: The findings highlight a critical gap: even when AI systems can be explained, stakeholders may not fully trust or agree with those explanations, raising questions about whether current XAI approaches adequately support responsible deployment of high-stakes prediction models.