Why AI Safety Requires Uncertainty, Incomplete Preferences, and Non-Archimedean Utilities

AI systems need to embrace uncertainty rather than assume they fully understand human values. The paper demonstrates that safe AI assistants must acknowledge what they don't know about human preferences and reason under that uncertainty, rather than confidently optimizing for potentially misunderstood goals.

Human preferences are often incomplete and contradictory, and AI systems must be designed to handle this reality. Rather than forcing AI to choose between conflicting human values, safe systems should be built to recognize when human preferences are unclear or in tension with each other, which is the normal state of human decision-making.

The "shutdown problem" is fundamental to AI safety. The paper identifies a critical challenge: safe AI systems must be willing to shut down when humans press a shutdown button, must not manipulate humans into pressing or not pressing it, and must still perform their intended tasks competently—requiring sophisticated reasoning about when to defer to human judgment.

Non-Archimedean utilities may be necessary for alignment. The paper suggests that traditional economic models of preference (where everything can be traded off on a single scale) may be insufficient; AI systems may need to reason about values that operate on different levels of importance that cannot be directly compared or traded against each other.

Current approaches to AI alignment may be mathematically incomplete. The formal framework presented suggests that existing methods for ensuring AI safety may be missing essential mathematical structures needed to properly handle the complexity of human values and the shutdown problem.