Mahsan Nourani, Chiradeep Roy, Tahrima Rahman, Eric D. Ragan, Nicholas Ruozzi, Vibhav Gogate
Inaccurate explanations can harm user performance more than providing no explanation at all. The study found that when an AI system provided low-veracity explanations for its decisions, users performed significantly worse on tasks compared to both receiving accurate explanations and receiving no explanations at all—suggesting that misleading transparency is counterproductive.
Simplifying AI models for interpretability can create a dangerous gap between explanations and actual decision-making. Designers often reduce the complexity of deep learning models to make them more understandable, but these simplified versions may generate explanations that don't accurately reflect how the actual model reached its conclusions, undermining the purpose of explainability.
Users may trust and rely on inaccurate explanations without realizing they're misleading. The research demonstrates that poor explanations can confuse users about why an AI system made a particular decision, potentially leading them to make worse choices based on false confidence in the system's reasoning.
Explanation quality directly impacts whether users find AI systems trustworthy and useful. The study shows that explanation veracity—whether the explanation truthfully represents the model's actual decision-making process—is critical to whether users will meaningfully engage with and benefit from explainable AI systems.
Responsible AI requires prioritizing accuracy over simplicity in explanations. Rather than defaulting to simplified explanations for the sake of user comprehension, developers should ensure explanations are first and foremost truthful to the model's actual behavior, as accuracy is more important than ease of understanding.
Explainable machine learning and artificial intelligence models have been used to justify a model's decision-making process. This added transparency aims to help improve user performance and understanding of the underlying model. However, in practice, explainable systems face many open questions and challenges. Specifically, designers might reduce the complexity of deep learning models in order to provide interpretability. The explanations generated by these simplified models, however, might not accurately justify and be truthful to the model. This can further add confusion to the users as they might not find the explanations meaningful with respect to the model predictions. Understanding how these explanations affect user behavior is an ongoing challenge. In this paper, we explore how explanation veracity affects user performance and agreement in intelligent systems. Through a controlled user study with an explainable activity recognition system, we compare variations in explanation veracity for a video review and querying task. The results suggest that low veracity explanations significantly decrease user performance and agreement compared to both accurate explanations and a system without explanations. These findings demonstrate the importance of accurate and understandable explanations and caution that poor explanations can sometimes be worse than no explanations with respect to their effect on user performance and reliance on an AI system.
Citation data from Semantic Scholar