"Let's Agree to Disagree": Investigating the Disagreement Problem in Explainable AI for Text Summarization

Different explanation methods often contradict each other: The paper demonstrates that when multiple explainability techniques are applied to the same AI summarization model, they frequently produce conflicting explanations about which parts of the text were most important to the model's decision—a widespread problem in current state-of-the-art systems that undermines user trust.

Disagreement between explanations reduces accountability: When different XAI methods yield inconsistent results, it becomes difficult for users, regulators, and stakeholders to confidently understand why an AI system made a particular decision, which is critical for responsible deployment of AI in high-stakes applications.

Breaking text into smaller segments improves explanation consistency: The proposed Regional Explainable AI (RXAI) approach divides articles into coherent segments before applying explanation methods, resulting in more consistent explanations across different XAI techniques compared to analyzing full texts at once.

The solution was validated on real-world summarization tasks: Testing on two major benchmark datasets (Xsum and CNN/Daily Mail) showed that the segmentation-based approach substantially decreased disagreement between explanation methods, suggesting practical applicability to deployed summarization systems.

Localized explanations enhance trustworthiness: By producing more consistent explanations through regional analysis, the approach helps foster greater confidence in AI-generated summaries, which is essential for users to safely rely on these systems in real-world applications.