AI2030 Responsible AI Tools Marketplace

Key Takeaways

Better explanations improve human understanding: A user study found that people could better understand and predict AI system decisions when shown "perception visualizations"—visual representations of what the AI actually perceives—compared to existing explanation methods like Grad-CAM, suggesting that how we explain AI matters significantly for trustworthiness.
Current AI explanation methods have limitations: Traditional explanation techniques used for images (perturbation and gradient-based methods) fail to fully capture the semantic information needed to explain why deep neural networks make their predictions, indicating a gap in our current ability to interpret these systems.
We can visualize what AI "sees": The paper introduces a new approach that reconstructs and displays the visual patterns that an AI model's internal representations correspond to, essentially showing what the neural network perceives in an image—providing a more intuitive window into AI decision-making.
Non-invasive explanations are possible: The visualization method works by inverting encoded features through a separate reconstruction model without modifying the original AI system's parameters or predictions, meaning explanations can be added to existing AI systems without changing how they operate.
Explainability supports responsible deployment: By making AI systems more understandable to humans, perception visualization can ease the debugging and deployment of deep learning models as "trusted systems," addressing a key concern in responsible AI about whether we can safely rely on these systems in real-world applications.

Abstract

Artificial intelligence (AI) systems power the world we live in. Deep neural networks (DNNs) are able to solve tasks in an ever-expanding landscape of scenarios, but our eagerness to apply these powerful models leads us to focus on their performance and deprioritises our ability to understand them. Current research in the field of explainable AI tries to bridge this gap by developing various perturbation or gradient-based explanation techniques. For images, these techniques fail to fully capture and convey the semantic information needed to elucidate why the model makes the predictions it does. In this work, we develop a new form of explanation that is radically different in nature from current explanation methods, such as Grad-CAM. Perception visualization provides a visual representation of what the DNN perceives in the input image by depicting what visual patterns the latent representation corresponds to. Visualizations are obtained through a reconstruction model that inverts the encoded features, such that the parameters and predictions of the original models are not modified. Results of our user study demonstrate that humans can better understand and predict the system's decisions when perception visualizations are available, thus easing the debugging and deployment of deep models as trusted systems.

Accepted paper at BMVC 2021 (Proceedings not available yet)

Key Takeaways

Better explanations improve human understanding: A user study found that people could better understand and predict AI system decisions when shown "perception visualizations"—visual representations of what the AI actually perceives—compared to existing explanation methods like Grad-CAM, suggesting that how we explain AI matters significantly for trustworthiness.
Current AI explanation methods have limitations: Traditional explanation techniques used for images (perturbation and gradient-based methods) fail to fully capture the semantic information needed to explain why deep neural networks make their predictions, indicating a gap in our current ability to interpret these systems.
We can visualize what AI "sees": The paper introduces a new approach that reconstructs and displays the visual patterns that an AI model's internal representations correspond to, essentially showing what the neural network perceives in an image—providing a more intuitive window into AI decision-making.
Non-invasive explanations are possible: The visualization method works by inverting encoded features through a separate reconstruction model without modifying the original AI system's parameters or predictions, meaning explanations can be added to existing AI systems without changing how they operate.
Explainability supports responsible deployment: By making AI systems more understandable to humans, perception visualization can ease the debugging and deployment of deep learning models as "trusted systems," addressing a key concern in responsible AI about whether we can safely rely on these systems in real-world applications.

Abstract

Accepted paper at BMVC 2021 (Proceedings not available yet)

Perception Visualization: Seeing Through the Eyes of a DNN

Key Takeaways

Abstract

Paper Info

Responsible AI Topics

Links

Perception Visualization: Seeing Through the Eyes of a DNN

Key Takeaways

Abstract

Paper Info

Responsible AI Topics

Links