Fixing confirmation bias in feature attribution methods via semantic match

Feature attribution methods can mislead users through confirmation bias: The paper demonstrates that visualizing which features a model uses isn't enough for humans to reliably understand what the model actually learned—people can easily convince themselves the model is working correctly when it isn't, a problem the authors call "confirmation bias in XAI."

There's a gap between what explanations show and what they mean: Simply showing that a model focuses on certain pixels or data points doesn't tell us whether the model is actually using human-understandable concepts. For example, a model might highlight relevant objects (good) or spurious correlations like background patterns (bad), and visual explanations alone can't distinguish between these cases.

A structured testing approach called "semantic match" is needed: Rather than casually interpreting feature attributions, the authors propose systematically testing whether human concepts actually match what the model learned, using specific metrics to measure this alignment across both tabular data and images.

The method can reveal both desirable and undesirable model behaviors: By applying semantic match assessment, researchers can identify when models appropriately focus on relevant objects for prediction versus when they rely on spurious correlations or other problematic patterns that explanations might otherwise hide.

This represents a first step toward more trustworthy AI explanations: The paper argues that structured semantic matching is necessary to move beyond the current limitations of feature attribution methods and build more reliable tools for understanding AI systems.