XALIGN: A Human-centric Visual Analytics Approach to Understanding AI Explanations

Different explanation methods often contradict each other: The paper found that rule-based explanations and feature attribution methods (TreeSHAP, KernelSHAP, and LIME) frequently disagreed when analyzing the same AI model predictions, highlighting that no single explanation method tells the complete story.

Comparing explanations side-by-side helps catch problems: By visualizing multiple explanation methods together, XALIGN revealed discrepancies that would have been missed if experts only looked at one method in isolation—a critical safeguard for high-stakes decisions like credit approval or weather prediction.

Visual comparison tools can prevent over-trust in AI: Expert evaluators found that seeing contradictions between explanation methods helped them avoid blindly trusting model predictions, suggesting that transparency tools are most effective when they show uncertainty and disagreement rather than false consensus.

Technical XAI expertise matters for interpretation: One expert without an XAI background found it harder to connect rule-based explanations to statistical attribution methods, indicating that even with better visualization tools, domain experts may need additional training to fully leverage explanation methods in practice.

Explanations need validation against real-world knowledge: The paper emphasizes that in high-stakes domains, experts must check AI explanations against established domain knowledge—visualization tools alone cannot substitute for expert judgment about whether an AI's reasoning makes sense.