Human Interpretation of Saliency-based Explanation Over Text

People misinterpret saliency explanations in predictable ways: The study found that laypeople often misunderstand which words are actually important according to AI models, with superficial factors like word length influencing their interpretation despite explanations communicating importance directly.

Current explanation methods don't guarantee user comprehension: While mathematical measures like gradient-based or Shapley value-based methods are mathematically sound, this research reveals a critical gap between what these explanations are designed to communicate and what people actually understand from them.

Visual presentation and input characteristics significantly affect interpretation: Various factors related to how explanations are visualized and presented, as well as properties of the input text itself, substantially influence how laypeople interpret saliency-based explanations—suggesting design choices matter greatly for responsible AI deployment.

Bias correction can improve explanation effectiveness: The researchers proposed a method to adjust saliencies based on estimates of how people over- or under-perceive certain features, demonstrating that some misinterpretation distortion can be reduced through thoughtful adjustment of explanation methods.

Explainability requires testing with actual users: The research emphasizes that responsible AI cannot assume explanations work as intended; empirical testing with real people across different languages and contexts is necessary to ensure explanations actually improve human understanding of AI decisions.