The Meta-Evaluation Problem in Explainable AI: Identifying Reliable Estimators with MetaQuantus

The XAI evaluation crisis is real: Different methods for evaluating explanation quality frequently produce conflicting rankings of the same explanation techniques, making it difficult for practitioners to know which explanation method actually works best.

Ground truth is the missing piece: A fundamental problem in XAI is that we often lack "ground truth" labels showing what a correct explanation should look like, making it nearly impossible to validate evaluation methods using traditional approaches.

MetaQuantus offers a practical solution: The framework evaluates quality estimators by testing two key properties—how well they resist noise and how they react to randomness—without requiring ground truth labels, providing a workaround to this validation problem.

Better evaluation tools enable better AI explanations: By helping practitioners identify which evaluation methods are most reliable, this work indirectly improves the quality of explanation methods themselves, since developers can now confidently select the best-performing techniques.

Open-source access democratizes XAI development: The authors released MetaQuantus as an open-source tool, allowing AI practitioners and researchers to benchmark and verify new evaluation methods, accelerating progress in making AI systems more transparent and trustworthy.