Rethinking XAI Evaluation: A Human-Centered Audit of Shapley Benchmarks in High-Stakes Settings

Quantitative metrics don't predict real-world usefulness: The paper found that standard measures used to evaluate explanation quality (like sparsity and faithfulness) don't actually correlate with whether explanations help humans understand decisions better or make better choices in practice.

Explanations may increase overconfidence rather than accuracy: Across 3,735 case reviews by professional analysts, explanations consistently boosted decision confidence but did not improve actual decision quality, raising concerns about "automation bias"—where people over-rely on AI systems because they feel more confident about them.

No single explanation method is clearly superior: Testing eight different Shapley value variants (competing formulations of a popular explainability technique) revealed no clear winner for improving analyst performance, suggesting the field's fragmentation into competing approaches may not reflect meaningful practical differences.

Current evaluation methods are misaligned with responsible AI deployment: The researchers demonstrate that the proxies typically used to benchmark explanation systems in academic settings fail to predict whether those systems will actually help or harm decision-making in high-stakes environments like fraud detection and risk assessment.

Human-centered testing is essential before deployment: The findings underscore that explainability methods require evaluation with real professionals in realistic operational settings—not just laboratory benchmarks—to ensure they support rather than undermine responsible decision-making in consequential domains.