Quality Metrics for Transparent Machine Learning With and Without Humans In the Loop Are Not Correlated

Human evaluation reveals what automated metrics miss: The study found that traditional XAI quality metrics (computed without human involvement) did not consistently rank interpretability methods the same way humans did, suggesting that automated assessments alone are insufficient for determining whether explanations actually help people understand AI systems.

Explanations must be tested with real users: Using psychophysical experiments with crowdsourced annotators, researchers discovered that the true usefulness of an explanation can only be authentically assessed through human-ML interaction tasks, not through computational analysis alone.

Practical impact matters more than technical sophistication: The research measured real outcomes like annotation accuracy and task completion time rather than relying solely on technical metrics, demonstrating that responsible AI evaluation should focus on whether explanations actually improve human decision-making in practice.

Current XAI evaluation methods may be misleading: The disconnect between automated quality metrics and human performance suggests that organizations deploying explainable AI systems could be making decisions based on metrics that don't reflect actual user benefit, potentially giving false confidence in system transparency.

Psychophysics offers a rigorous framework for AI evaluation: By adapting established experimental methods from psychology, the researchers demonstrated a more robust approach to assessing interpretability that could become a standard practice for evaluating whether transparency features genuinely serve their intended purpose.