The Role of Explanation Styles and Perceived Accuracy on Decision Making in Predictive Process Monitoring

Explanation alone isn't enough—perceived accuracy matters more: The study found that how accurate users believe the AI system to be significantly influences their decision-making, sometimes more than the explanation itself. This suggests that building user trust requires communicating AI reliability clearly, not just providing technical explanations.

Different explanation styles work differently for different decisions: The research tested three explanation approaches (feature importance, rule-based, and counterfactual explanations) and found they had varying effects on task performance and user confidence. Organizations implementing explainable AI should consider which explanation style best fits their specific use case rather than assuming one approach works universally.

Users need both objective performance metrics and subjective confidence measures: The study measured not just whether users made correct decisions (task performance) but also how confident they felt in those decisions. This distinction is important because overconfident users making wrong decisions can be more problematic than underconfident users who double-check their work.

Current XAI evaluations miss what actually matters for users: The paper highlights that most explainable AI research focuses on technical metrics like "fidelity" rather than real-world outcomes like whether explanations actually help people make better decisions. This gap means many deployed XAI systems may look good on paper but fail to improve human decision-making in practice.