Scrutinizing XAI using linear ground-truth data with suppressor variables

Most popular explanation methods fail basic tests: When researchers tested eight common explainability methods (including LIME, SHAP, and others) on a carefully controlled dataset, most were unable to reliably distinguish between features that actually influence predictions and "suppressor variables" that don't—raising concerns about their trustworthiness in real-world applications.

Explainability tools can highlight irrelevant information: The paper demonstrates that saliency methods (tools designed to show which data inputs matter most) can point to features that have no statistical connection to the prediction target, potentially misleading decision-makers who rely on these explanations.

There's no agreed-upon definition of "feature importance": A fundamental problem in the field is that explainable AI lacks a formal, universally accepted definition of what "feature importance" actually means, making it difficult to validate whether explanation methods are working correctly.

Ground-truth testing reveals hidden problems: By creating a dataset where all relationships between data and outcomes are precisely known and linear, researchers uncovered failures in explanation methods that might go undetected in real-world scenarios, suggesting these tools need more rigorous validation before use in high-stakes decisions.

Responsible AI deployment requires better validation standards: The findings suggest that before using explainability methods to justify important decisions (in healthcare, criminal justice, lending, etc.), organizations should demand more rigorous testing and clearer definitions of what these tools actually measure.