A Taxonomy for Design and Evaluation of Prompt-Based Natural Language Explanations

NLEs are becoming critical governance tools: As large language models become more prevalent, Natural Language Explanations (generated through prompts) are now essential for helping stakeholders understand how AI systems work, making them a key focus for responsible AI oversight and accountability.

A comprehensive framework exists for evaluating explanations: The paper provides a structured taxonomy across three dimensions—Context (who needs the explanation and why), Generation and Presentation (how it's created and shown), and Evaluation (how to assess quality)—that researchers, auditors, and policymakers can use to systematically design and verify AI explanations.

One-size-fits-all explanations won't work: The Context dimension emphasizes that effective explanations must be tailored to specific tasks, audiences, and goals; an explanation suitable for a technical auditor may not work for a consumer or regulator, highlighting the need for customized approaches to AI transparency.

How explanations are presented matters as much as their content: The Generation and Presentation dimension identifies that factors like interactivity, output format, and presentation method significantly influence whether stakeholders can actually understand and trust the explanation—not just whether the information is accurate.

Evaluation of explanations requires multiple approaches: The taxonomy calls for assessing explanations across content quality, presentation clarity, and user-centered properties (like whether people actually understand them), suggesting that no single evaluation method can fully determine if an explanation serves its governance purpose.