Large Language Models Cannot Explain Themselves

LLM "explanations" don't reflect how models actually work: Large language models can be prompted to produce explanations of their outputs, but these explanations do not accurately represent the mechanical processes that generated the predictions in the first place, creating a false sense of understanding.

The illusion of transparency poses real risks: The appearance that these explanations reflect genuine reasoning can cause significant harms, as users may trust and act on explanations that don't actually describe the model's decision-making process.

Reframing explanations as "exoplanations" highlights their limitations: The paper proposes using the term "exoplanations" to emphasize that these outputs are external to and disconnected from the model's actual mechanisms, rather than true explanations of its reasoning.

These outputs have value for critical thinking, not model understanding: While LLM-generated explanations shouldn't be trusted as accurate descriptions of how the model works, they can still be useful for prompting human critical evaluation and questioning of the model's outputs.

Design safeguards are necessary: The paper recommends implementing guardrails and specific responses when models are prompted to generate explanations, suggesting that system design should acknowledge and mitigate the risks posed by misleading explanations.