Back to the Baseline: Examining Baseline Effects on Explainability Metrics

Current methods for evaluating AI explanations are biased by an arbitrary choice: The paper reveals that popular metrics used to test whether AI explanation methods work (Insertion and Deletion metrics) depend heavily on a "baseline"—a reference point chosen by researchers. Different baseline choices favor different explanation methods, meaning comparisons between methods may not be fair or reliable.

There's a fundamental trade-off in how we evaluate explanations: Existing baselines face a dilemma—they either successfully remove information from images (which is necessary for fair testing) or they avoid creating unrealistic, distorted images. The paper found that no current baseline achieves both goals simultaneously, suggesting the evaluation process itself has built-in limitations.

Even simple AI models can contradict themselves about which explanation method is best: When researchers tested a basic linear model using common baselines, it identified different "optimal" explanation methods depending on which baseline was used. This inconsistency raises serious questions about whether current evaluation practices can reliably tell us which explanation methods actually work best.

A new approach shows promise for fairer evaluation: The researchers developed a novel baseline that better balances the competing demands—it removes information effectively while keeping generated images more realistic and representative of what the model typically sees, potentially providing more trustworthy comparisons between explanation methods.

This work highlights why standardization matters for responsible AI: The findings underscore that technical choices in evaluation metrics, even seemingly minor ones like baseline selection, can significantly impact which AI explanation methods we trust and deploy, emphasizing the need for careful standardization in XAI research.