VirtualXAI: A User-Centric Framework for Explainability Assessment Leveraging GPT-Generated Personas

XAI evaluation needs to go beyond technical metrics: Current methods for assessing explainability focus primarily on quantitative measures like fidelity and consistency, but the paper demonstrates that qualitative factors—such as user satisfaction and actual interpretability—are equally important for determining whether an AI explanation is truly useful to people.

Different users need different explanations: The framework uses AI-generated personas representing diverse user backgrounds and expertise levels to test explanations. This reflects a key insight: a technical explanation that satisfies a data scientist may confuse a business manager or end user, so responsible AI requires tailoring explanations to the audience.

Practitioners need practical guidance for selecting AI tools: The paper identifies a significant gap where organizations struggle to choose which datasets, AI models, and explanation methods work best for their specific situations. The proposed recommender system addresses this by matching new problems to previously benchmarked solutions, potentially democratizing access to responsible AI practices.

Combining human judgment with automated assessment improves trustworthiness: By integrating both quantitative benchmarking and qualitative user feedback through virtual personas, the framework provides a more holistic evaluation of whether an AI system can genuinely be trusted—moving beyond purely technical validation toward human-centered assessment.