A Study of Automatic Metrics for the Evaluation of Natural Language Explanations

Embedding-based metrics outperform traditional word-matching methods: The study found that newer automatic evaluation measures like BERTScore and BLEURT correlate better with human judgments of explanation quality than older metrics like BLEU and ROUGE, suggesting that how we measure AI transparency needs to evolve beyond simple word-overlap comparisons.

Automatic evaluation of explanations is now feasible: By creating the ExBAN corpus of crowd-sourced natural language explanations for Bayesian Networks, the researchers demonstrated that we can systematically evaluate whether AI-generated explanations are actually good—an important step toward holding AI systems accountable for their transparency claims.

Human judgment remains the gold standard: The research validates that human ratings of explanations are meaningful and measurable, establishing a benchmark against which automatic evaluation methods can be tested—this matters because it confirms that "good explanations" isn't just a subjective notion but something we can objectively assess.

Transparency evaluation deserves dedicated research: By drawing parallels between explanation evaluation and the established field of Natural Language Generation evaluation, the paper highlights that assessing AI transparency requires specialized methods rather than borrowing generic language evaluation tools.