F-Fidelity: A Robust Framework for Faithfulness Evaluation of Explainable AI

Existing XAI evaluation methods have a fundamental flaw: The most common approach to test whether AI explanations are accurate—removing important features and checking if predictions change—creates unrealistic data that doesn't match real-world patterns, making the evaluation unreliable.

A popular "fix" to that problem creates a new one: The ROAR method addresses the unrealistic data issue by retraining models, but this approach inadvertently leaks information about the explanations themselves into the evaluation process, leading to unfair and potentially misleading comparisons between different explanation methods.

F-Fidelity offers a more balanced solution: The proposed framework uses a neutral fine-tuning approach that doesn't favor any particular explanation method, combined with random masking techniques that keep test data realistic, addressing both the original distribution problem and the information leakage issue simultaneously.

The framework was rigorously tested across multiple domains: The researchers validated F-Fidelity using state-of-the-art explanation methods on images, time series data, and natural language—demonstrating that the framework works reliably across different types of AI applications, not just one specific use case.

Better evaluation methods matter for trustworthy AI: By establishing more rigorous and fair ways to evaluate whether AI explanations are actually faithful to how models make decisions, this work helps ensure that organizations can confidently rely on XAI techniques for high-stakes decisions.