Why is "Problems" Predictive of Positive Sentiment? A Case Study of Explaining Unintuitive Features in Sentiment Classification

Explainable AI systems can produce confusing results that mislead users: The paper demonstrates that machine learning models can identify patterns that seem backwards to humans—like the word "problems" being predictive of positive sentiment in product reviews—highlighting a gap between what models learn and what users intuitively expect.

Standard explanations aren't enough for responsible AI: Simply telling users "this feature predicts this outcome" can leave them confused or mistrustful if the association seems illogical. The research shows that additional context is needed to help users understand why counterintuitive patterns exist in model predictions.

Detecting and explaining unintuitive patterns is feasible: The paper proposes methods to automatically identify when a model's predictive features might confuse users, and then generate explanations for those features. A crowdsourced study with 300 participants confirmed these approaches effectively help people understand puzzling model behavior.

This addresses a real gap in responsible AI practice: The authors note that explaining unintuitive associations between inputs and predictions is "underexplored" in XAI research, suggesting that many current AI explanation tools may be leaving users confused without realizing it—a potential trust and safety issue.

User understanding matters for AI accountability: By focusing on what confuses actual people rather than just technical accuracy, this work emphasizes that responsible AI requires explanations tailored to human comprehension, not just algorithmic transparency.