ExplainabilityTransparencyAccountabilitycs.CV
Improving the Explain-Any-Concept by Introducing Nonlinearity to the Trainable Surrogate Model
Mounes Zaval, Sedat Ozer
May 20, 2024
Signal Processing and Communications Applications Conference
Key Takeaways
-
Better explanations require more sophisticated models: The paper demonstrates that adding a nonlinear layer to explanation models improves their ability to accurately represent how AI systems make decisions, suggesting that oversimplified explanation methods may miss important aspects of how neural networks actually work.
-
Automatic concept detection is advancing: By building on the Segment Anything Model's ability to automatically identify visual concepts, this research moves away from relying on human-annotated concepts, making AI explanations more scalable and potentially less biased by human labeling choices.
-
Improvements are measurable across real-world datasets: The researchers validated their approach on both ImageNet and MS COCO datasets, indicating that these explanation improvements aren't just theoretical but work on diverse, practical computer vision tasks that AI systems encounter in real applications.
-
Concept-based explanations offer a human-friendly alternative: Rather than just highlighting individual pixels, this work supports concept-based approaches that explain AI decisions in terms of recognizable visual elements, which may be more understandable to non-technical stakeholders evaluating AI system trustworthiness.
Abstract
In the evolving field of Explainable AI (XAI), interpreting the decisions of deep neural networks (DNNs) in computer vision tasks is an important process. While pixel-based XAI methods focus on identifying significant pixels, existing concept-based XAI methods use pre-defined or human-annotated concepts. The recently proposed Segment Anything Model (SAM) achieved a significant step forward to prepare automatic concept sets via comprehensive instance segmentation. Building upon this, the Explain Any Concept (EAC) model emerged as a flexible method for explaining DNN decisions. EAC model is based on using a surrogate model which has one trainable linear layer to simulate the target model. In this paper, by introducing an additional nonlinear layer to the original surrogate model, we show that we can improve the performance of the EAC model. We compare our proposed approach to the original EAC model and report improvements obtained on both ImageNet and MS COCO datasets.