Do We Need Another Explainable AI Method? Toward Unifying Post-hoc XAI Evaluation Methods into an Interactive and Multi-dimensional Benchmark

There are too many explainable AI methods and practitioners need help choosing them: The paper notes that the number of published XAI algorithms has "underged a boom," making it difficult for data scientists to select appropriate tools for their specific needs.

XAI methods are frequently misused and misinterpreted: The research highlights that "data scientists could misuse xAI algorithms and misinterpret their results," suggesting that having an explanation method alone is insufficient without proper guidance on correct usage.

A unified benchmark can standardize how we evaluate XAI methods: The paper proposes Compare-xAI, which consolidates multiple testing approaches into one framework organized around five key evaluation categories: fidelity, fragility, stability, simplicity, and stress tests.

Different users need different information about XAI methods: The benchmark is designed with three hierarchical levels tailored to different audiences—researchers, practitioners, and non-experts—recognizing that explanations of XAI performance should match the audience's technical expertise and needs.

Standardized evaluation could improve responsible AI deployment: By providing a structured way to assess whether XAI methods actually work as intended and can be correctly interpreted, this benchmark addresses a gap between the legal requirement for explanations (mentioned as a right in various countries) and the practical ability to deliver trustworthy explanations.