Synthetic Benchmarks for Scientific Research in Explainable Machine Learning

Explainability tools like LIME and SHAP are widely used but hard to evaluate: The paper identifies a critical gap in the field—while feature attribution methods have become popular for explaining AI decisions, there's no standardized way to test whether they actually work well, making it difficult for organizations to choose which tool to trust.

Human studies and real-world testing are impractical at scale: Current evaluation methods either require expensive human studies or are computationally prohibitive on real datasets, creating a barrier to rigorous testing of explainability methods before they're deployed in high-stakes applications.

Synthetic datasets enable faster, more rigorous benchmarking: XAI-Bench uses configurable synthetic datasets to efficiently compute ground-truth explanations, allowing researchers to systematically evaluate and compare different explainability techniques without the computational burden of real-world data.

Standardized benchmarking can accelerate responsible AI deployment: By providing a shared library and benchmark suite, this work helps bridge the gap between developing explainability methods and actually deploying them responsibly, enabling organizations to validate that their AI explanations are reliable before use.

Customizable testing parameters support diverse real-world scenarios: The synthetic datasets can be configured to simulate various data characteristics, allowing researchers to test how well explainability methods perform across different conditions they might encounter in practice.