Evaluating the Ability of Explanations to Disambiguate Models in a Rashomon Set

Multiple models can make identical predictions while using completely different reasoning: The paper demonstrates that within a "Rashomon set" (a group of models with similar performance), different models can reach the same conclusions through different decision-making processes. This means that two AI systems with identical accuracy could be fundamentally different in how they work, highlighting why explanations matter beyond just performance metrics.

Standard explanation evaluation methods can be fooled into validating false reasoning: The research shows that conventional approaches to checking if explanations are "correct" can be manipulated. Models can be deliberately designed to make accurate predictions while providing misleading explanations, and traditional evaluation methods fail to catch this deception—a problem the authors call "adversarial fairwashing."

The proposed AXE method successfully detects when AI systems secretly use protected characteristics: Unlike existing evaluation approaches, AXE can identify when models are using sensitive attributes (like race or gender) to make decisions, even when those attributes aren't mentioned in the explanations. The paper reports 100% success in detecting this type of hidden bias.

Choosing the right explanation evaluation method is critical for responsible AI deployment: Because different explainers can produce different explanations for the same model, organizations need robust evaluation methods to ensure they're selecting trustworthy models for real-world use. The paper suggests that current industry-standard approaches may be insufficient for this task.