XAI-Arena: Can LLMs Assess the Quality of XAI Explanations?

LLMs can reliably assess XAI explanation quality: The study found a strong positive correlation (Spearman's rho=.693, p<.001) between LLM-generated ratings and human expert ratings, suggesting that large language models can serve as a reproducible alternative to subjective human evaluation of AI explanations.

Current XAI evaluation relies too heavily on subjective judgment: The paper identifies a critical gap in responsible AI practice—existing methods for evaluating whether AI explanations are actually good depend on inconsistent human judgment, which limits reproducibility and makes it difficult to compare different explanation methods fairly across studies.

XAI-Arena enables multidimensional evaluation across stakeholder needs: The framework assesses explanations across eight dimensions (simplicity, clarity, task adequacy, trust calibration, actionability, transparency, faithfulness, and interpretability) and can be customized for different stakeholder personas, recognizing that different users may need different types of explanations.

Scalability and reproducibility are now achievable for XAI assessment: By using LLMs as judges, organizations can evaluate explanation quality consistently across many datasets and machine learning models without the time and cost constraints of recruiting human evaluators, making responsible AI practices more practical to implement at scale.

Systematic comparison of XAI methods is now possible: The framework enables fair benchmarking of different explanation techniques, helping practitioners identify which methods produce higher-quality explanations for their specific use cases and stakeholder needs.