ExplainabilityTransparencyTrustworthinesscs.LG
EMaP: Explainable AI with Manifold-based Perturbations
Minh N. Vu, Huy Q. Mai, My T. Thai
September 18, 2022
Journal of machine learning research
3 citations
Key Takeaways
-
Better explanations require understanding data structure: EMaP improves explanation methods by perturbing data along directions that preserve the underlying structure (manifold) of the data, rather than making random changes. This means explanations become more faithful to how the model actually makes decisions.
-
Robustness against manipulation: The paper demonstrates that EMaP-based explanations can resist recently-developed attacks designed to fool explanation methods, suggesting that this approach provides more reliable and trustworthy insights into AI model behavior that cannot be easily gamed.
-
Mathematical rigor improves real-world reliability: By using formal mathematical measures (Gromov-Hausdorff distance and persistent homology) to validate that perturbations preserve data topology, the researchers provide theoretical grounding that translates to more dependable explanations in practice—important for high-stakes applications.
-
Perturbation-based explanations matter for accountability: Since many explanation methods work by slightly changing inputs to see how model outputs change, improving this fundamental technique has broad implications for making black-box AI systems more interpretable and accountable across different industries and applications.
Abstract
In the last few years, many explanation methods based on the perturbations of input data have been introduced to improve our understanding of decisions made by black-box models. The goal of this work is to introduce a novel perturbation scheme so that more faithful and robust explanations can be obtained. Our study focuses on the impact of perturbing directions on the data topology. We show that perturbing along the orthogonal directions of the input manifold better preserves the data topology, both in the worst-case analysis of the discrete Gromov-Hausdorff distance and in the average-case analysis via persistent homology. From those results, we introduce EMaP algorithm, realizing the orthogonal perturbation scheme. Our experiments show that EMaP not only improves the explainers' performance but also helps them overcome a recently-developed attack against perturbation-based methods.