Explaining Learned Reward Functions with Counterfactual Trajectories

Learned reward functions—systems trained to understand human values—often fail to extract correct rewards, highlighting a critical gap between what AI systems learn and what humans actually want, making interpretability tools essential for catching these flaws before deployment.

The paper proposes Counterfactual Trajectory Explanations (CTEs), which work by showing how changing specific actions in a scenario changes the reward the AI assigns, allowing humans to understand the reasoning behind the reward function through concrete before-and-after comparisons rather than abstract rules.

The method was validated by training a proxy-human model on CTEs, which successfully learned to judge reward differences between trajectories and generalized to new, out-of-distribution examples, suggesting this explanation approach could help humans evaluate whether learned reward functions actually align with human values.

While CTEs don't provide perfect understanding of reward functions, the authors frame this as a realistic starting point, acknowledging that complete transparency may be unattainable but that incremental improvements in interpretability are valuable for responsible AI deployment.

This work demonstrates that adapting existing explainability techniques from AI safety research to reward learning is a "fruitful approach," suggesting that combining interpretability methods with value alignment could be a practical path forward for making AI systems more trustworthy and auditable.