Embodied Explainability and Ontological Obstacles: Why We Struggle to Explain the Answers of Large Language Models (LLMs)

Explainability is not just a technical property to extract from AI systems. The paper argues that understanding LLM outputs is created through interaction and shared practice between humans and the system, not simply by looking at the model's internal workings. This means effective explanations require active engagement rather than passive information display.

Current explanation methods may be fundamentally misnamed and misleading. The authors identify "ontological obstacles" when researchers try to peer inside LLMs—the explanations they generate often reflect external frameworks imposed by researchers rather than how the model actually works. This mismatch can increase user overreliance on explanations that feel authoritative but may not accurately represent the system's reasoning.

Focusing solely on internal model mechanisms misses a critical part of the explanation process. The paper emphasizes that explainers themselves shape understanding through their choices and interpretations. Responsible AI requires acknowledging that explanations are co-created between the system, the explainer, and the user—not objective truths extracted from the model.

True explainability should enable users to probe, coordinate, and repair AI behavior in real-world contexts. Rather than providing static explanations, responsible AI systems should offer affordances (opportunities for action) that let people understand what matters in their specific situation and take corrective action when needed.

We should be more cautious about claiming something is "explainable." The paper suggests reserving explainability claims for designs that genuinely support situated practice and user agency, rather than applying the label to any system that provides internal model information.