No Single Metric Tells the Whole Story: A Multi-Dimensional Evaluation Framework for Uncertainty Attributions

No single metric can fully evaluate uncertainty attribution methods: The study found that while most metrics rank methods consistently across samples, "inter-method agreement remains low," indicating that researchers and practitioners need to assess uncertainty explanations using multiple complementary metrics rather than relying on one evaluation approach.

Gradient-based methods outperform perturbation-based approaches for explaining uncertainty: The research demonstrates that "gradient-based methods consistently outperform perturbation-based approaches in consistency and conveyance," suggesting that the technical approach used to attribute uncertainty to input features significantly impacts explanation quality.

A new "conveyance" property is essential for uncertainty-specific evaluation: The framework introduces conveyance as a property specifically designed for uncertainty attributions, which "evaluates whether controlled increases in epistemic uncertainty reliably propagate to feature-level attributions"—addressing a gap in existing XAI evaluation frameworks that weren't designed with uncertainty in mind.

Standardized evaluation frameworks improve comparability across XAI research: By aligning uncertainty attribution evaluation with the established Co-12 framework and proposing concrete implementations for properties like correctness, consistency, continuity, and compactness, this work addresses the problem that "studies rely on heterogeneous proxy tasks and metrics, hindering comparability" across different research efforts.

Different uncertainty quantification methods have varying strengths: The experiments show that "Monte-Carlo dropconnect outperforms Monte-Carlo dropout in most metrics," suggesting that the choice of uncertainty quantification technique meaningfully affects how well uncertainty can be explained to users.