Towards a Unified Multidimensional Explainability Metric: Evaluating Trustworthiness in AI Models

Explainability isn't one-size-fits-all: The research shows that how well an AI explanation method works depends on the specific dataset, the type of AI model being used, and who's trying to understand it. This means organizations can't simply pick one explanation tool and assume it will work equally well across all their AI systems.

Three key qualities matter for trustworthy explanations: The framework identifies fidelity (how accurately the explanation reflects what the model actually does), simplicity (how easy it is to understand), and stability (how consistent the explanations are). According to the paper, evaluating AI systems on all three dimensions—rather than just one—provides a more complete picture of trustworthiness.

A practical tool for comparing explanation methods: The research creates a knowledge base that can predict how well popular explanation techniques like LIME and SHAP will perform on new datasets and models without having to test them from scratch. This could help organizations more quickly choose the right explanation method for their specific AI applications.

Context matters for evaluation: The paper emphasizes that the "best" explanation method varies based on context—including dataset characteristics and user expertise. This suggests that responsible AI deployment requires tailoring explanation approaches rather than applying generic solutions across different use cases.