In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Making

Explanations alone don't improve decision-making: The paper finds that AI explanations "rarely enable complementary performance," meaning that simply explaining how an AI system reached its conclusion doesn't consistently help humans make better decisions when working alongside AI.

Verifiability is the missing link: The authors argue that explanations only help when they allow humans to actually verify whether the AI's prediction is correct. Many current explanation methods fail at this fundamental task, focusing instead on interpretability or describing reasoning without enabling verification.

Many real-world tasks are inherently difficult to verify: The paper identifies a structural problem: "most tasks fundamentally do not allow easy verification, regardless of explanation method," meaning that even well-designed explanations cannot overcome the challenge that humans cannot independently confirm whether an AI's answer is right.

"Appropriate reliance" is more nuanced than assumed: The research distinguishes between two types of appropriate reliance—"outcome-graded" (trusting the AI when it's likely correct) and "strategy-graded" (trusting the AI for the right reasons)—suggesting that current studies may be measuring success in oversimplified ways.

Current AI explanation research may be solving the wrong problem: The findings suggest that the field's focus on making AI systems more interpretable or transparent may not address what humans actually need: concrete ways to verify AI predictions before acting on them.