Evaluating Explanations Through LLMs: Beyond Traditional User Studies

LLMs show promise as a scalable alternative to traditional user studies: The paper demonstrates that Large Language Models can replicate most conclusions from human-based studies evaluating explainable AI tools, potentially reducing the time and cost barriers that currently limit XAI evaluation in critical sectors like healthcare.

Not all LLMs perform equally in replicating human judgment: The research found that "different LLMs yield varying levels of alignment in the results," meaning organizations cannot assume all language models will produce consistent or reliable evaluations—model selection matters significantly.

Technical factors like memory and variability impact reliability: The study identified that "experimental factors such as LLM memory and output variability affect alignment with human responses," suggesting that LLM-based evaluation requires careful attention to how these systems are configured and may not be universally reliable without proper controls.

This approach addresses a real bottleneck in responsible AI development: By offering a faster, cheaper way to evaluate explainability tools, this method could help organizations more rapidly test whether their AI systems are actually transparent and trustworthy to users—a key component of responsible AI deployment.

Human validation remains necessary: While the findings are "initial," the varying performance across different LLMs and the acknowledged gaps in alignment suggest that LLM-based evaluation should complement rather than replace human studies when high-stakes decisions are involved.