Challenges and Opportunities in Text Generation Explainability

Explainability for AI text generation is harder than it seems: The paper identifies 17 distinct challenges in explaining how large language models generate text, spanning technical issues like tokenization and stochastic (random) behavior, making it difficult to reliably understand what the model is doing at each step.

Current evaluation methods are inadequate: There is no consensus on how to measure whether an explanation is actually good or similar to other explanations, and no standard metrics for determining which parts of the input truly matter for the model's output—meaning we may be using flawed methods to assess AI transparency.

Human judgment is essential but underutilized: The paper emphasizes that involving humans throughout the entire explainability process—from designing test datasets to evaluating explanations—is critical for drawing reliable conclusions, yet this human-in-the-loop approach remains underdeveloped in current practice.

We need better tools designed for how language models actually work: Most explainability methods treat text generation as a simple process, but autoregressive models (which generate one word at a time with randomness built in) require specialized, probabilistic approaches that account for this complexity.

Standardized testing is lacking: The absence of suitable benchmark datasets and test cases makes it difficult to compare different explainability methods fairly, hindering progress toward more trustworthy and interpretable AI systems.