Subgoal-Based Explanations for Unreliable Intelligent Decision Support Systems

AI systems make mistakes, and users need to know it: The research challenges a critical assumption in prior work that AI decision support systems always provide correct recommendations. By studying how users interact with systems that occasionally fail or give wrong advice, the authors highlight that real-world AI systems are "not perfect" and may produce "incorrect output or fail to work altogether."

Explaining the "why" helps users catch AI errors: Subgoal-based explanations—which show users what intermediate objective an AI recommendation is trying to achieve—enabled users to "distinguish optimal and suboptimal IDS recommendations" better than traditional explanations. This means users were more capable of recognizing when the AI system was steering them wrong.

Better explanations improve human performance even when AI fails: The study found that subgoal-based explanations led to "improved user task performance" and "more robust user performance in the case of IDS failure." This suggests that when AI systems break down or become unavailable, users who received better explanations were better equipped to continue performing their tasks independently.

Users prefer explanations that show reasoning steps: The research demonstrates that subgoal-based explanations were "preferred by users," indicating that people want to understand not just what an AI recommends, but the logical reasoning chain behind that recommendation.

Explainability is essential for responsible AI deployment: This work underscores that responsible AI decision support requires more than accurate recommendations—it requires transparent explanations that help users maintain appropriate skepticism and retain the ability to make sound decisions when AI systems fail.