Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems

Proxy tasks don't measure real-world performance: The paper found that when researchers use artificial stand-in tasks to evaluate explainable AI systems—such as asking humans to predict an AI's decisions based on explanations—these results do not actually predict how well humans and AI will perform together on real decision-making tasks. This suggests the field may be optimizing for the wrong metrics.

User trust and preference are unreliable indicators of actual performance: Subjective measures like whether users trust the AI system or prefer one explanation style over another did not correlate with objective performance on actual tasks. People may feel confident in an AI system that doesn't actually help them make better decisions, or vice versa.

Current evaluation methods may be slowing progress: By relying on these misleading evaluation approaches, the XAI field risks developing systems that appear to work well in laboratory settings but fail to improve human-AI team performance in practice, potentially delaying the development of genuinely effective collaborative AI systems.

Real-world evaluation is essential: The paper demonstrates the importance of testing explainable AI systems through actual decision-making tasks rather than simplified proxies, suggesting that responsible AI development requires evaluation methods that reflect how these systems will actually be used.