ExplainabilityTransparencyAccountabilitycs.LG
Explanatory Masks for Neural Network Interpretability
Lawrence Phillips, Garrett Goh, Nathan Hodas
November 15, 2019
arXiv.org
5 citations
Key Takeaways
-
Explanation masks can reveal what neural networks actually focus on: The method creates visual or textual "masks" that highlight the specific parts of an input (pixels in images, words in text, molecular features in chemicals) that the network uses to make its predictions, making black-box decisions more transparent.
-
The approach works across different types of AI systems: The researchers successfully demonstrated their technique on three distinct applications—image classification, sentiment analysis, and chemical property prediction—suggesting the method has broad applicability rather than being limited to one domain.
-
Explanations can be made minimal without sacrificing accuracy: By training a secondary network to find the smallest possible explanation while maintaining the original network's predictive performance, the method avoids over-explaining and focuses on truly essential features, which could help identify potential biases or spurious correlations.
-
This enables post-hoc analysis of already-deployed models: Since the method works on pre-trained networks, organizations can apply it to existing AI systems without retraining them from scratch, making interpretability more practical for real-world deployment scenarios where models are already in production.
Abstract
Neural network interpretability is a vital component for applications across a wide variety of domains. In such cases it is often useful to analyze a network which has already been trained for its specific purpose. In this work, we develop a method to produce explanation masks for pre-trained networks. The mask localizes the most important aspects of each input for prediction of the original network. Masks are created by a secondary network whose goal is to create as small an explanation as possible while still preserving the predictive accuracy of the original network. We demonstrate the applicability of our method for image classification with CNNs, sentiment analysis with RNNs, and chemical property prediction with mixed CNN/RNN architectures.