Assessing Fidelity in XAI post-hoc techniques: A Comparative Study with Ground Truth Explanations Datasets

Not all AI explanation methods are equally reliable: The study found that some popular XAI techniques perform poorly at accurately explaining what their underlying AI models actually do, highlighting the need to evaluate explanation methods before trusting them in real-world applications.

Backpropagation-based explanations outperform alternatives: Methods that trace information backward through neural networks produced more accurate explanations compared to sensitivity analysis or Class Activation Maps (CAM), suggesting researchers should prioritize these approaches when developing trustworthy AI systems.

Ground truth datasets are essential for accountability: By creating datasets with verified correct explanations, the researchers demonstrated a practical way to objectively test XAI methods—addressing a major gap in the field where explanation accuracy was previously difficult to measure.

Trade-offs exist between accuracy and clarity: While backpropagation methods proved most accurate, they tended to produce noisier, harder-to-interpret visualizations, suggesting that achieving both reliable and understandable AI explanations requires careful consideration of these competing demands.

Eliminating unreliable methods advances responsible AI: The study's goal of identifying and removing low-fidelity explanation techniques from further development could help prevent misleading explanations from being deployed in high-stakes domains like healthcare or criminal justice.