Brittle AI, Causal Confusion, and Bad Mental Models: Challenges and Successes in the XAI Program

One-size-fits-all explanations don't work: The paper finds that explanations must be "highly tailored to their scenario," meaning that a single explanation method cannot effectively serve all users and contexts. This suggests organizations deploying AI systems need to customize how they explain decisions based on who needs the information and why.

High performance doesn't mean reliable AI: The research reveals that "many seemingly high performing RL agents are extremely brittle," meaning AI systems that appear to work well in testing can fail unpredictably in real-world conditions. This challenges the assumption that good benchmark performance translates to trustworthy systems ready for deployment.

Causal explanations are powerful but complicated: While causal models enable "rich explanations" of why AI systems make decisions, the paper notes that "how to present them isn't always straightforward." This highlights a gap between having technical understanding and being able to communicate it effectively to decision-makers and affected parties.

People develop incorrect mental models of AI that are hard to correct: The paper's most striking finding is that "human subjects conjure fantastically wrong mental models for AIs, and these models are often hard to break." This suggests that even when organizations provide explanations, people may misunderstand how AI actually works, potentially leading to inappropriate trust or misuse of these systems.