Uncertainty Propagation in XAI: A Comparison of Analytical and Empirical Estimators

XAI methods can produce unreliable explanations when uncertainty isn't properly accounted for. The paper identifies specific XAI methods that fail to reliably capture and propagate uncertainty from data and model parameters, which is particularly concerning for high-stakes applications like healthcare or criminal justice where explanations inform critical decisions.

Uncertainty in AI explanations comes from two main sources: noisy input data and variability in model parameters. The framework shows that explanations themselves can vary significantly depending on small changes to the data fed into a model or slight adjustments to how the model is trained—a finding that challenges the assumption that explanations are stable and trustworthy.

Both mathematical and experimental approaches to measuring explanation uncertainty have different strengths. The paper compares analytical estimates (theoretical calculations) with empirical estimates (real-world testing) of how uncertainty flows through explanations, suggesting that practitioners should validate explanations using multiple methods rather than relying on a single approach.

Current XAI methods have significant limitations that users should be aware of. The comprehensive evaluation across different datasets reveals that not all popular explanation techniques handle uncertainty consistently, meaning organizations deploying XAI tools should test whether their chosen method reliably produces trustworthy explanations under real-world conditions.

Uncertainty-aware explanations are essential for building genuine trust in AI systems. Rather than presenting explanations as definitive, the paper argues that showing confidence levels and uncertainty ranges in explanations would help decision-makers better understand when to rely on AI recommendations and when to seek additional information.