Trust in AI: Interpretability is not necessary or sufficient, while black-box interaction is necessary and sufficient

Black-box interaction matters more than interpretability: The paper finds that the ability to repeatedly test and interact with an AI system (running it "at-will") is both necessary and sufficient for building trust, while interpretability alone is neither necessary nor sufficient—challenging the popular assumption that explainable AI is the key to trustworthiness.

Trust requires evidence of consistent behavior, not explanations: The authors propose using "behavior certificates" that compile evidence from multiple sources—including real-world testing, out-of-distribution evaluations, and theoretical proofs—to predict and verify how an AI will actually perform, rather than relying on post-hoc explanations of its decisions.

Interpretability has hidden costs: While interpretability can sometimes help build trust, the paper cautions that it can also create false confidence or misleading explanations, suggesting that organizations should carefully weigh the benefits against potential downsides rather than treating interpretability as an automatic solution.

Different trust contexts require different approaches: The paper distinguishes between "human-AI trust" (trusting a tool directly) and "human-AI-human trust" (trusting an AI system used by others), implying that one-size-fits-all trust solutions are inadequate for responsible AI deployment.

Current AI evaluation processes are insufficient: The authors argue that existing methods for assessing AI trustworthiness are confused and inadequate, with major implications for how AI is deployed in high-stakes domains like science, healthcare, and public policy.