From Plausible to Actionable: A Position on LLM Self-Explanations

Self-explanations from LLMs sound convincing but may not reflect how the model actually works. The paper distinguishes between "plausible" explanations (ones that sound reasonable to humans) and "faithful" ones (ones that accurately represent the model's actual reasoning), noting that LLMs often excel at the former while their faithfulness remains uncertain.

Current methods for evaluating AI explanations are insufficient for LLMs. The authors identify limitations in standard explainability evaluation protocols, suggesting that existing approaches designed for other AI systems don't adequately assess whether LLM explanations are trustworthy or just convincing-sounding.

Usefulness to decision-makers should be a key evaluation criterion alongside accuracy. Beyond asking whether explanations are faithful, the paper argues we should evaluate whether explanations actually help stakeholders (users, regulators, developers) make better-informed decisions and take appropriate action—a criterion often overlooked in traditional AI explainability research.

Organizations need practical guidelines for assessing LLM explanations. The authors propose concrete guidelines for evaluating both plausibility and faithfulness, offering a path forward for practitioners who need to determine whether they can rely on LLM-generated explanations in real-world applications.

LLM explanations can be valuable even with uncertainty about their faithfulness. The paper suggests that self-explanations have legitimate uses for supporting decision-making and transparency, provided stakeholders understand their limitations and don't assume the explanations necessarily reveal the model's true reasoning process.