Feature salience - not task-informativeness - drives machine learning model explanations

XAI methods may highlight visually prominent features rather than decision-relevant ones: The study found that five popular attribution methods assigned substantially elevated importance to watermarks in images (R² ≥ .45) regardless of whether those watermarks actually informed the model's predictions, suggesting that visual salience—not informativeness—drives importance scores.

Current XAI tools may fail to detect shortcut learning: Because attribution methods prioritize salient features over task-informative ones, they cannot reliably identify when models rely on spurious correlations or shortcuts rather than meaningful patterns, undermining a key promise of explainable AI for catching model failures.

The presence of confounding information matters less than its visual properties: Whether watermarks were class-dependent (actually useful for prediction) or class-independent noise had only marginal effects on importance attribution (R² ≤ .03), indicating that how features look matters more to XAI methods than whether they're actually predictive.

XAI methods behave like edge detectors, not true explainers: The finding that importance attribution depends on image intensity encoding (bright vs. dark features) suggests popular XAI methods may function similarly to basic image processing filters rather than genuinely explaining model reasoning.

Users should be cautious about trusting XAI explanations at face value: These results indicate that feature importance scores from standard XAI tools may be misleading, potentially giving false confidence in model decisions or causing practitioners to overlook actual model failures.