Expectation Alignment: Handling Reward Misspecification in the Presence of Expectation Mismatch

Misspecified objectives are a fundamental AI safety problem that lacks clear definition: The paper identifies that while AI researchers recognize reward misspecification as critical to safety, there has been no formal framework defining what constitutes a misspecified objective or how to know when it's been successfully resolved—a gap this work attempts to address.

Human expectations about AI behavior are key to understanding objective misalignment: The framework uses "theory of mind" (understanding what humans believe the AI will do) as the foundation for detecting misspecification, suggesting that AI safety requires explicitly modeling the gap between what users expect and what the system actually does.

Current methods for handling reward misspecification have significant limitations: The paper reveals that existing approaches to fixing misspecified rewards are incomplete, and the Expectation Alignment framework provides concrete insights into why these methods fall short and what new strategies could work better.

Interactive algorithms can infer user expectations from specified rewards: Rather than requiring users to perfectly specify objectives upfront, the proposed method uses an interactive approach to work backward from the stated reward function to understand what users actually expect the system to do, making AI systems more adaptable to human intent.

The solution is computationally tractable for practical implementation: By mapping the inference problem into linear programs, the authors demonstrate that their approach can be efficiently implemented on standard AI benchmarks, suggesting practical applicability beyond theoretical frameworks.