Local Explanations for Reinforcement Learning

RL policies are harder to explain than classification models: While explainable AI research has extensively focused on explaining what classification models do, deep reinforcement learning policies—which make sequential decisions over time—have received far less attention despite their increasing real-world deployment in robotics and autonomous systems.

Grouping states by expert behavior improves human understanding: Rather than grouping similar actions together, the researchers' approach identifies "meta-states" based on how an expert policy naturally moves through a problem space. A user study demonstrated this perspective leads to better human understanding compared to previous explanation methods.

Important states reveal interpretable intermediate goals: The key insight is that identifying critical decision points within these meta-states helps humans understand policies as sequences of tractable intermediate goals rather than opaque black-box decisions, making RL systems more transparent to domain experts.

The approach is theoretically sound and computationally efficient: The algorithm for finding important states is mathematically proven to converge, and uses submodular optimization to ensure high-quality explanations can be selected efficiently—meaning explanations can be generated practically for real systems.

Tested across multiple domains with human validation: The method was validated not just on benchmark environments (four rooms, Pac-Man, Pong) but through a carefully designed user study, suggesting the findings about interpretability have genuine practical value beyond theoretical performance metrics.