Predictability and Comprehensibility in Post-Hoc XAI Methods: A User-Centered Analysis

Popular explanation tools have significant comprehensibility gaps: The study found that SHAP, one of the most widely used explainability methods, becomes significantly harder for users to understand when explaining predictions for cases near the model's decision boundary—precisely where explanations matter most for high-stakes decisions.

Counterfactual explanations substantially improve user understanding: The research demonstrates that showing users "what would need to change" for a different prediction (counterfactual explanations) significantly increases their ability to understand how machine learning models make decisions, suggesting this approach should be prioritized in AI systems affecting people's lives.

Showing model mistakes builds better mental models: When users were shown examples of misclassifications, their understanding of how the model works improved significantly, indicating that transparency about failure cases is crucial for responsible AI deployment rather than hiding errors.

Current explainability methods don't guarantee predictability: Even with explanations from tools like LIME and SHAP, users struggled to predict how models would behave in new situations, revealing a gap between having explanations and actually understanding model behavior—a critical concern for trustworthy AI systems.

Design matters for responsible AI: The paper provides specific design recommendations for future explainability tools, emphasizing that how explanations are presented directly affects whether non-technical users can meaningfully understand and appropriately trust AI systems making decisions about them.