ExplainabilityTransparencySafetycs.LG
Improving Explainability of Softmax Classifiers Using a Prototype-Based Joint Embedding Method
Hilarie Sit, Brendan Keith, Karianne Bergen
July 2, 2024
arXiv.org
Key Takeaways
-
Explainability through examples: The method allows AI systems to explain their decisions by pointing to specific training examples (prototypes) that influenced the prediction, making it easier for users to understand why a model made a particular decision rather than just accepting a confidence score.
-
More reliable confidence measures: Traditional softmax classifiers often express high confidence even on unfamiliar data. This approach provides more trustworthy uncertainty estimates by measuring how similar new inputs are to known training examples, which is particularly important for detecting when a model encounters data it wasn't trained on.
-
Better detection of unfamiliar inputs: The method demonstrates improved ability to identify "out of distribution" data—inputs that differ significantly from training examples. This is crucial for responsible AI deployment, as it helps prevent systems from making confidently wrong predictions on novel or adversarial inputs.
-
Instance-based reasoning over black boxes: By grounding predictions in actual training examples rather than abstract learned features, this approach moves toward more interpretable AI systems that stakeholders can audit and verify, supporting accountability in high-stakes applications.
Abstract
We propose a prototype-based approach for improving explainability of softmax classifiers that provides an understandable prediction confidence, generated through stochastic sampling of prototypes, and demonstrates potential for out of distribution detection (OOD). By modifying the model architecture and training to make predictions using similarities to any set of class examples from the training dataset, we acquire the ability to sample for prototypical examples that contributed to the prediction, which provide an instance-based explanation for the model's decision. Furthermore, by learning relationships between images from the training dataset through relative distances within the model's latent space, we obtain a metric for uncertainty that is better able to detect out of distribution data than softmax confidence.