A Comparative User Evaluation of XRL Explanations using Goal Identification

Most explainable AI systems for game-playing agents fail basic tests: When researchers asked users to identify an agent's goal from XRL explanations, only one out of four algorithms performed better than random guessing, suggesting current explanation methods may not effectively communicate what AI systems are actually trying to do.

Users can't accurately judge their own understanding: The study found a critical disconnect—users reported feeling confident they understood the agent's goals and found explanations easy to follow, but their actual accuracy didn't match these self-assessments, highlighting a potential risk where people may overestimate how well they truly comprehend AI decision-making.

Current evaluation methods for explainable AI are inadequate: The paper demonstrates that simply asking users if they "understand" an explanation isn't sufficient; more rigorous testing (like goal identification tasks) is needed to verify that explanations actually work as intended before deploying AI systems in real-world applications.

Debugging AI systems requires better explanation tools: Since debugging is a critical use case for explainable AI—helping developers identify and fix problems—the poor performance of current XRL algorithms suggests significant work remains before these tools can reliably help humans maintain and improve AI systems.