The Importance of Distrust in AI

Blind trust in AI is as problematic as distrust: The paper argues that exclusively focusing on making AI systems more "trustworthy" can backfire, as users may overtrust AI capabilities. The example of AI chatbots that "tend to make up facts" despite appearing coherent illustrates how high trust without appropriate skepticism can lead to harmful reliance on flawed systems.

Distrust serves a protective function: Rather than viewing distrust as purely negative, the paper identifies "a healthy amount of distrust" as valuable for ensuring users don't over-rely on AI systems. This distinction between trust and distrust is presented as essential for achieving "appropriate reliance" on AI.

The explainability solution has limitations: While explainability methods are widely promoted as ways to increase user trust in AI, the paper suggests this approach may be incomplete. An exclusive focus on transparency and trust-building may not adequately address the need for users to maintain critical judgment about AI limitations.

Appropriate reliance requires balancing two risks: The paper identifies that both "disuse" (refusing to use helpful AI) and "overtrust" (relying too heavily on flawed AI) are problems that need mitigation. Achieving the right balance requires users to understand both what AI systems can do well and where they are fallible.

Current AI development may encourage inappropriate trust: The paper suggests that recent advances in AI coherence and capability presentation may inadvertently promote overtrust, making it increasingly important to cultivate user skepticism alongside understanding of AI systems.