When concept-based XAI is imprecise: Do people distinguish between generalisations and misrepresentations?

People struggle to recognize when AI generalizations are actually beneficial: The study found that when an AI system showed generalized explanations (varying irrelevant details like what people were doing) to demonstrate it had learned the core concept (people's position relative to tracks), users rated these explanations as worse than precise matches, suggesting they couldn't distinguish helpful generalization from poor performance.

Imprecision in AI explanations can undermine trust regardless of intent: Explanations that generalized over less important features received ratings no better than explanations that systematically misrepresented those same features, indicating that users penalize any deviation from exact matching without understanding whether the AI's reasoning is actually sound.

Current concept-based explanations may not effectively communicate AI reasoning to non-experts: The research reveals a gap between what makes AI systems work well (generalizing beyond specific details) and what makes explanations feel trustworthy to people (precise, literal matching), suggesting that concept-based explanations need redesign to help users understand when variation in explanations reflects good AI behavior.

Safety-critical domains require clearer explanation design: In the railway safety scenario tested, users' difficulty distinguishing good generalizations from poor ones raises concerns about whether people can properly evaluate AI systems used in high-stakes decisions affecting public safety.