It Takes Two to Tango: Towards Theory of AI's Mind

Humans need to understand AI systems, not just vice versa. The paper argues that effective human-AI teamwork requires a two-way street: while AI should understand human intentions and contexts, humans must also develop a "theory of AI's mind" by learning the system's strengths, weaknesses, and failure patterns—a concept traditionally overlooked in AI research.

People can quickly learn to predict AI failures with minimal training. The study found that lay people (non-experts) could be trained with just 50 examples to better predict when a Visual Question Answering model would fail, suggesting that understanding AI behavior is learnable and doesn't require deep technical expertise.

Explainability tools may not be as helpful as expected. Surprisingly, the paper found that providing humans with the model's internal states—such as confidence scores in its top predictions—did not necessarily improve their ability to understand and predict the AI system's behavior, raising questions about which explanation methods actually help people build mental models of AI.

Better human-AI collaboration could improve team performance. Drawing on research showing that teams with strong theory of mind capabilities demonstrate higher collective performance, the paper suggests that helping humans understand AI systems could lead to more effective collaboration and better real-world outcomes.