When more precision is worse: Do people recognize inadequate scene representations in concept-based explainable AI?

People may not recognize when AI systems rely on irrelevant information: The study found that participants actually preferred or didn't mind when the AI retained irrelevant features (like background scenery) in its explanations, suggesting they failed to recognize this as a flaw in the model's decision-making process.

Current XAI explanations may create false confidence: Participants rated the AI more favorably when it retained relevant features, but this positive assessment didn't extend to recognizing problems with irrelevant feature use—meaning people may develop unwarranted trust in AI systems based on incomplete understanding of how they actually work.

The gap between technical correctness and human understanding is significant: Even when XAI tools technically show which features an AI uses (through concept images), non-technical users struggle to interpret whether those features are actually appropriate for the decision being made, highlighting a critical limitation in current explanation methods.

Responsible AI deployment requires more than showing explanations: Simply providing visual explanations of AI decisions isn't sufficient to help people identify flawed reasoning; new approaches may be needed to help users recognize when AI systems are using spurious correlations or irrelevant information to reach conclusions.