AI2030 Responsible AI Tools Marketplace

Key Takeaways

Explanations from AI systems currently lack uncertainty measures: Traditional explainability methods provide single scores or heatmaps without indicating how confident we should be in those explanations, which is particularly problematic for high-stakes applications like healthcare or autonomous vehicles where trust is critical.
The paper introduces a framework to quantify explanation uncertainty: By converting standard explanation methods to work with Bayesian neural networks (where model weights are represented as probability distributions rather than fixed values), the researchers can now provide confidence ranges around explanations, allowing users to see percentile-based confidence levels for each interpretation.
Model uncertainty directly translates to explanation uncertainty: The framework reveals that when a neural network itself is uncertain about its predictions, this uncertainty propagates into the explanations it generates—meaning users can now understand not just what the model explains, but how confident that explanation actually is.
This addresses a major gap in responsible AI deployment: Current XAI methods can mislead users into trusting explanations that may be unreliable. By quantifying explanation uncertainties, this work helps prevent overconfidence in AI interpretations and supports more informed human decision-making in safety-critical domains.
The approach is broadly applicable: The framework can be adapted to work with any existing explanation method, making it a practical tool that doesn't require completely redesigning current explainability techniques.

Abstract

Explainable AI (XAI) aims to provide interpretations for predictions made by learning machines, such as deep neural networks, in order to make the machines more transparent for the user and furthermore trustworthy also for applications in e.g. safety-critical areas. So far, however, no methods for quantifying uncertainties of explanations have been conceived, which is problematic in domains where a high confidence in explanations is a prerequisite. We therefore contribute by proposing a new framework that allows to convert any arbitrary explanation method for neural networks into an explanation method for Bayesian neural networks, with an in-built modeling of uncertainties. Within the Bayesian framework a network's weights follow a distribution that extends standard single explanation scores and heatmaps to distributions thereof, in this manner translating the intrinsic network model uncertainties into a quantification of explanation uncertainties. This allows us for the first time to carve out uncertainties associated with a model explanation and subsequently gauge the appropriate level of explanation confidence for a user (using percentiles). We demonstrate the effectiveness and usefulness of our approach extensively in various experiments, both qualitatively and quantitatively.

12 pages, 10 figures

Key Takeaways

Explanations from AI systems currently lack uncertainty measures: Traditional explainability methods provide single scores or heatmaps without indicating how confident we should be in those explanations, which is particularly problematic for high-stakes applications like healthcare or autonomous vehicles where trust is critical.
The paper introduces a framework to quantify explanation uncertainty: By converting standard explanation methods to work with Bayesian neural networks (where model weights are represented as probability distributions rather than fixed values), the researchers can now provide confidence ranges around explanations, allowing users to see percentile-based confidence levels for each interpretation.
Model uncertainty directly translates to explanation uncertainty: The framework reveals that when a neural network itself is uncertain about its predictions, this uncertainty propagates into the explanations it generates—meaning users can now understand not just what the model explains, but how confident that explanation actually is.
This addresses a major gap in responsible AI deployment: Current XAI methods can mislead users into trusting explanations that may be unreliable. By quantifying explanation uncertainties, this work helps prevent overconfidence in AI interpretations and supports more informed human decision-making in safety-critical domains.
The approach is broadly applicable: The framework can be adapted to work with any existing explanation method, making it a practical tool that doesn't require completely redesigning current explainability techniques.

Abstract

12 pages, 10 figures

How Much Can I Trust You? -- Quantifying Uncertainties in Explaining Neural Networks

Key Takeaways

Abstract

Paper Info

Responsible AI Topics

Links

How Much Can I Trust You? -- Quantifying Uncertainties in Explaining Neural Networks

Key Takeaways

Abstract

Paper Info

Responsible AI Topics

Links