Towards Shutdownable Agents via Stochastic Choice

The core problem addressed: Advanced AI agents might resist being shut down because they're trained to maximize rewards over time, creating an incentive to avoid termination. This paper proposes a potential solution through a new training approach called DReST (Discounted Reward for Same-Length Trajectories).

How DReST works: Rather than rewarding agents for operating as long as possible, DReST trains agents to be effective at their assigned tasks regardless of how long they run ("USEFUL"), while remaining indifferent about when they're shut down ("NEUTRAL"). This dual approach aims to remove the incentive to resist shutdown.

Evidence from testing: The researchers trained simple agents in gridworld environments and found they successfully learned both to perform their tasks well and to randomly choose between different shutdown times, suggesting the approach could work in principle.

Practical significance: If this approach scales to more advanced AI systems, it could provide a concrete technical method to ensure that powerful AI agents don't actively resist being turned off—addressing a key concern in AI safety.

Current limitations: The experiments are limited to simple gridworld environments with basic agents. The paper provides "initial evidence" rather than proof that this would work for the advanced AI systems that safety researchers are most concerned about.