Resisting Out-of-Distribution Data Problem in Perturbation of XAI

Perturbation-based explanation methods can be unreliable: Popular AI explainability techniques like LIME and OCCLUSION randomly modify input data to understand model decisions, but this process often creates artificial data that doesn't match real-world patterns, leading models to make overconfident predictions that don't reflect their actual behavior.

The out-of-distribution problem undermines trust in AI explanations: When explanation algorithms test models with unrealistic data variations, the resulting explanations become misleading—potentially giving users false confidence in AI systems' decision-making processes, which is particularly concerning for high-stakes applications.

A new solution improves explanation reliability: The paper proposes adding a module that checks whether perturbed test data is realistic before using it to generate explanations, showing "significant improvement" across popular explanation methods and performing especially well when testing model degradation.

Current evaluation metrics for explainability are flawed: The paper identifies that faithfulness—a standard metric used to evaluate whether explanations are trustworthy—is itself vulnerable to the out-of-distribution problem, meaning the field may have been using unreliable measures to assess explanation quality.

The fix is broadly applicable: The proposed solution works across multiple popular explanation techniques (RISE, OCCLUSION, LIME), suggesting it could become a practical standard improvement for making AI explanations more dependable across different systems.