Confident Teacher, Confident Student? A Novel User Study Design for Investigating the Didactic Potential of Explanations and their Impact on Uncertainty

AI explanations don't always improve accuracy over predictions alone: In a study of 1,200 participants performing complex biological species annotation tasks, users shown AI explanations were not significantly more accurate than those shown only the model's prediction, suggesting explanations may not provide the practical benefit many assume.

Explanations can increase over-reliance on AI, even when wrong: Users were more likely to simply copy the AI model's predictions when explanations were provided, even when those predictions were incorrect—a concerning finding for responsible AI deployment where blind trust in explained decisions could propagate errors.

AI assistance reduces uncertainty but doesn't teach lasting skills: While users felt more confident and showed less uncertainty when working with AI assistance, they did not retain improved annotation abilities after the collaborative task ended, indicating explanations may create a false sense of understanding rather than genuine learning.

Human-centered evaluation of explanations matters: The study demonstrates the importance of testing XAI systems with real users on realistic tasks rather than relying solely on automated metrics, as the actual impact on human decision-making and learning differed from what offline evaluation might predict.

Context-specific design is critical for responsible XAI: The findings suggest that explanations must be carefully designed for their intended purpose—whether to improve immediate task performance, prevent over-reliance, or enable learning—rather than assuming one explanation approach works universally.