EXACT: Towards a platform for empirically benchmarking Machine Learning model explanation methods

Popular explanation methods may not be as reliable as assumed: The EXACT platform found that many widely-used XAI methods "often struggle to surpass random baselines," meaning they sometimes perform no better than randomly identifying important features—a critical finding for organizations relying on these tools to understand their AI systems.

Explanation quality varies significantly across similar models: The research demonstrates "variability in explanations derived from different equally performing model architectures," indicating that two AI models with identical accuracy can produce different explanations for the same decision, raising concerns about consistency and trustworthiness.

Current XAI methods have a fundamental problem with irrelevant features: The platform revealed that explanation methods frequently attribute significance to "irrelevant features," potentially misleading users about what factors actually drive AI decisions—a serious issue for high-stakes applications like healthcare or criminal justice.

We need standardized testing for explanation methods: EXACT provides "a standardised foundation for evaluating XAI methods" with benchmark datasets and quantitative metrics, addressing the challenge that XAI evaluation has historically lacked formal standards—similar to how we test other software before deployment.

Explainability research requires empirical validation: The paper highlights that XAI is "an inherently unsupervised process" that has faced challenges in "formalisation and empirical validation," suggesting that many explanation methods have been adopted without rigorous testing of whether they actually work as intended.