Feature relevance quantification in explainable AI: A causal problem

Current explainability tools may have theoretical flaws: The popular SHAP software package, widely used to explain AI decisions, relies on conditional probability distributions that the authors argue are conceptually incorrect for measuring feature importance, though some practical implementations happen to use the right approach anyway.

The distinction between observation and intervention matters: When determining which features matter for an AI decision, there's a crucial difference between asking "what do we observe when a feature is missing?" versus "what would happen if we deliberately removed a feature?" — the paper argues the field has conflated these two fundamentally different questions.

Causality frameworks are essential for responsible AI: Drawing on Pearl's foundational work on causality, the authors demonstrate that properly explaining AI decisions requires careful causal reasoning, not just statistical correlation, which has implications for how we validate and trust AI systems.

"Improvements" to explainability tools may backfire: Some recent attempts to enhance SHAP's feature relevance quantification may actually move away from the theoretically correct approach, potentially making explanations less reliable even if they appear more sophisticated.

Theoretical rigor in AI explanation is still developing: This paper highlights that even popular, widely-adopted explainability methods lack settled theoretical foundations, suggesting the field should prioritize getting the conceptual basics right before deploying these tools for high-stakes decisions.