Why the Valuable Capabilities of LLMs Are Precisely the Unexplainable Ones

The most useful AI capabilities may be fundamentally difficult to explain: The paper argues that what makes LLMs valuable—their ability to handle nuanced, context-dependent tasks—exists precisely in the parts we cannot translate into simple human-readable rules. This challenges the assumption that better AI means more interpretable AI.

Expert systems failed for a reason: The paper uses historical evidence that expert systems (AI systems built on explicit rules) consistently underperformed compared to modern LLMs, suggesting that rule-based approaches have inherent limitations for complex real-world problems that LLMs overcome through their "unexplainable" capabilities.

There's a fundamental mismatch between how humans explain things and how complex systems work: The paper argues that human-readable rules are poorly suited for capturing how LLMs actually solve problems, drawing on the Chinese philosophical concept of "Wu" (learning through practice rather than explicit instruction) to suggest some knowledge cannot be fully articulated.

This creates a genuine tension for AI safety and accountability: If the most valuable capabilities of LLMs cannot be fully explained through rules, this has serious implications for responsible AI development—it becomes harder to audit, predict, or control these systems, raising questions about how we can ensure they're safe and trustworthy.

We may need to rethink what "understanding" an AI system means: Rather than pursuing complete explainability, the paper suggests the field should grapple with whether other forms of verification, testing, and oversight might be more appropriate for systems whose power comes from their irreducible complexity.