"What Is It That You Don't Understand?" Language Games and Black Box Algorithms

"Interpretability" is more honest than "explainability": The paper argues we should stop claiming we can fully explain AI systems and instead acknowledge we can only achieve partial understanding. This distinction matters because it sets realistic expectations about what transparency efforts can actually deliver to users and regulators.

The reference problem is fundamental: Drawing on philosopher Quine's thought experiment, the paper shows that AI systems face an inherent problem similar to a linguist trying to understand a foreign word without full context—we can never be completely certain what an algorithm is actually "referring to" or basing its decisions on, even with explanation tools.

Complete transparency is impossible: The paper concludes that the hope for full algorithmic transparency is "undoubtedly in vain." Instead of pursuing an unattainable goal, organizations should focus on developing "partial and broad interpretations" that acknowledge their limitations rather than claiming comprehensive understanding.

Language games, not universal rules: Following Wittgenstein, the paper suggests there's no single correct way to interpret or explain algorithms. Different stakeholders (regulators, users, developers) may need different types of explanations for different purposes, rather than one universal explanation.