Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences

AI researchers may be designing explanations for themselves rather than actual users. The paper argues that explainable AI (XAI) risks the same problem that plagued software design: technical experts building systems based on their own needs and understanding rather than the needs of the people who will actually use them.

Explanations need to be grounded in how humans actually think and learn. The paper emphasizes that insights from psychology, cognitive science, and philosophy should guide XAI development, rather than purely technical approaches. This means understanding human reasoning, memory limitations, and how people naturally process information.

Success of explainable AI depends more on evaluating human understanding than on technical metrics. Rather than measuring whether an AI system can produce explanations, researchers should focus on whether real users actually understand those explanations and can use them effectively to make decisions or trust the system appropriately.

There is a significant gap between current XAI research and established knowledge from social and behavioral sciences. The paper identifies considerable untapped potential to incorporate existing research from philosophy, psychology, and cognitive science into how AI explanations are designed and evaluated.

Explainable AI requires interdisciplinary collaboration beyond computer science. To build truly useful AI explanations, researchers need to work with and learn from experts in human behavior, cognition, and communication rather than treating explanation as purely a technical problem.