ExplainabilityBiasTransparencycs.SE
Test Automation with Grad-CAM Heatmaps -- A Future Pipe Segment in MLOps for Vision AI?
Markus Borg, Ronald Jabangwe, Simon Åberg, Arvid Ekblom, Ludwig Hedlund, August Lidfeldt
March 2, 2021
International Conference on Software Testing, Verification and Validation Workshops
20 citations
Key Takeaways
-
Explainability tools like Grad-CAM can help make AI systems more trustworthy: The paper demonstrates that Grad-CAM heatmaps—visual explanations showing which parts of an image a model focuses on—can help meet the EU's seven key requirements for Trustworthy AI, making opaque deep learning models more transparent for critical applications like autonomous vehicles.
-
Automated testing can catch hidden biases in AI models: The authors propose integrating heatmap analysis into MLOps pipelines to automatically detect when models are making decisions based on irrelevant or "invalid pixels" rather than appropriate features, which could reveal biased or unreliable models before deployment.
-
Visual explanations are particularly important for safety-critical AI: The paper emphasizes that opaqueness in large ML models is a "substantial impediment for critical applications such as in the automotive context," suggesting that explainability tools are essential when AI systems affect public safety.
-
Explainability should be built into AI development workflows: Rather than treating explainability as an afterthought, the authors advocate for adding automated heatmap analysis as a standard component of MLOps pipelines, suggesting that responsible AI requires systematic processes, not just individual tools.
Abstract
Machine Learning (ML) is a fundamental part of modern perception systems. In the last decade, the performance of computer vision using trained deep neural networks has outperformed previous approaches based on careful feature engineering. However, the opaqueness of large ML models is a substantial impediment for critical applications such as in the automotive context. As a remedy, Gradient-weighted Class Activation Mapping (Grad-CAM) has been proposed to provide visual explanations of model internals. In this paper, we demonstrate how Grad-CAM heatmaps can be used to increase the explainability of an image recognition model trained for a pedestrian underpass. We argue how the heatmaps support compliance to the EU's seven key requirements for Trustworthy AI. Finally, we propose adding automated heatmap analysis as a pipe segment in an MLOps pipeline. We believe that such a building block can be used to automatically detect if a trained ML-model is activated based on invalid pixels in test images, suggesting biased models.