Evaluating XAI Support From A Hierarchical Reinforcement Learning Policy in Human-Agent Collaboration

Explanations alone don't automatically improve task performance: In this study of human-agent collaboration in a cooking simulation, participants who received real-time explanations from the AI agent showed no significant difference in overall performance compared to those without explanations, challenging the assumption that more transparency always leads to better outcomes.

How explanations are delivered matters as much as what they say: Audio explanations significantly damaged participants' sense of partnership with the agent, while text explanations did not produce this effect. The researchers suggest this occurred because hearing spoken explanations created expectations of a more capable, responsive partner than the underlying AI could actually provide.

Explanation systems must match the AI's actual capabilities: The study reveals a critical mismatch problem—when an AI delivers explanations through a medium (like natural speech) that implies a higher level of partnership capacity than it can actually sustain, it can backfire and harm human-agent collaboration. This suggests responsible XAI design requires careful alignment between how explanations are presented and what the AI can realistically deliver.

Benchmark testing of learned AI policies is essential for real-world applicability: Previous XAI research relied on hand-crafted policies in custom environments, limiting practical relevance. This study provides the first systematic evaluation of explanations from a learned AI policy in an established benchmark, establishing methodology that could improve how XAI systems are tested before deployment.