Calibrated Explanations: with Uncertainty Information and Counterfactuals

Explanations need uncertainty estimates: The paper addresses a critical gap in AI explainability by showing that feature importance explanations should include uncertainty information—not just point estimates. This means when an AI system explains why it made a decision, it should also communicate how confident it is in that explanation.

Model calibration improves trustworthiness: The research demonstrates that poorly calibrated machine learning models produce unreliable feature importance explanations. By using calibration techniques (Venn-Abers), the method ensures that when a model says it's 80% confident, it actually is correct about 80% of the time, making explanations more trustworthy for real-world decisions.

Instability in explanations is a real problem: The paper identifies that feature importance explanations are often unstable and unreliable, which is particularly concerning when these explanations influence high-stakes decisions. The proposed method addresses this by providing stable, robust explanations across different scenarios.

Counterfactual explanations with uncertainty are practical: Beyond explaining what happened, the method generates counterfactual explanations (e.g., "if feature X had been different, the outcome would have changed") while quantifying uncertainty. This helps users understand not just why a decision was made, but what would need to change and how confident we should be in that alternative scenario.

The solution works across different AI models: Because the method is model-agnostic, it can be applied to various types of AI systems without requiring redesign, making it potentially useful for improving explainability across many real-world applications.