ExplainabilityAccountabilityTransparencycs.CY
Making Sense of the Unsensible: Reflection, Survey, and Challenges for XAI in Large Language Models Toward Human-Centered AI
Francisco Herrera
May 18, 2025
arXiv.org
15 citations
Francisco Herrera
Explainability is critical for high-stakes decisions: As LLMs are deployed in sensitive domains like healthcare, law, and education, the paper emphasizes that transparent and interpretable AI systems are essential for ensuring these models can be trusted and held accountable when they influence important decisions affecting people's lives.
Four key dimensions define good explanations: The paper identifies faithfulness, truthfulness, plausibility, and contrastivity as core requirements for LLM explanations. This means explanations must accurately reflect how the model actually works, provide accurate information, be understandable to users, and help people understand why the model chose one answer over another.
Different stakeholders need different explanations: The research highlights that explainability must be "audience-sensitive," meaning doctors, lawyers, regulators, and patients each need explanations tailored to their specific roles and decision-making needs—a one-size-fits-all approach to transparency won't work in practice.
Current transparency efforts often fall short: The paper shifts focus from "surface-level transparency" to "governance-ready design," suggesting that simply showing how an AI system works isn't enough; explanations must actually support regulatory compliance and enable responsible oversight in real-world deployment.
Mechanistic interpretability and causal reasoning represent emerging solutions: The paper identifies newer technical approaches—understanding the internal mechanisms of LLMs and incorporating causal reasoning—as promising directions for creating explanations that are both technically sound and genuinely useful for human decision-makers.
As large language models (LLMs) are increasingly deployed in sensitive domains such as healthcare, law, and education, the demand for transparent, interpretable, and accountable AI systems becomes more urgent. Explainable AI (XAI) acts as a crucial interface between the opaque reasoning of LLMs and the diverse stakeholders who rely on their outputs in high-risk decisions. This paper presents a comprehensive reflection and survey of XAI for LLMs, framed around three guiding questions: Why is explainability essential? What technical and ethical dimensions does it entail? And how can it fulfill its role in real-world deployment? We highlight four core dimensions central to explainability in LLMs, faithfulness, truthfulness, plausibility, and contrastivity, which together expose key design tensions and guide the development of explanation strategies that are both technically sound and contextually appropriate. The paper discusses how XAI can support epistemic clarity, regulatory compliance, and audience-specific intelligibility across stakeholder roles and decision settings. We further examine how explainability is evaluated, alongside emerging developments in audience-sensitive XAI, mechanistic interpretability, causal reasoning, and adaptive explanation systems. Emphasizing the shift from surface-level transparency to governance-ready design, we identify critical challenges and future research directions for ensuring the responsible use of LLMs in complex societal contexts. We argue that explainability must evolve into a civic infrastructure fostering trust, enabling contestability, and aligning AI systems with institutional accountability and human-centered decision-making.
30 pages, 1 figure
Citation data from Semantic Scholar