On the consistent reasoning paradox of intelligence and optimal trust in AI: The power of 'I don't know'

Consistent reasoning creates an unavoidable trade-off: AI systems that reason consistently (handling equivalent questions phrased differently, like "What is the time?" and "Tell me the time!") will inevitably make mistakes and produce plausible-sounding but false answers—a phenomenon called hallucination—infinitely often, even on basic tasks like arithmetic.

"I don't know" is essential for trustworthy AI: The paper argues that any AI system that both reasons consistently and aims to be trustworthy must be able to say "I don't know" rather than always providing an answer. This capability is currently missing from modern AI systems and requires developing what the authors call an "'I don't know' function."

Detecting hallucinations is harder than solving problems: The research demonstrates that identifying when an AI has hallucinated is mathematically more difficult than solving the original problem itself, meaning we cannot easily verify whether an AI's answers are correct even after the fact.

Perfect explanations aren't always possible: There are problems where an AI can produce the correct answer but cannot provide a logically sound explanation for how it arrived at that answer, highlighting a fundamental limitation in AI transparency and interpretability.

Human-like intelligence requires human-like fallibility: The paper establishes that achieving human-level reasoning in AI necessarily means accepting human-level mistakes—there is no way to have sophisticated reasoning without the possibility of error.