EthicsSafetyAccountabilitycs.AI
Toward a Theory of Value in AI Alignment
Andrew Smart, Shazeda Ahmed, Jackie Kay, Jimmy Tobin, Kris Shrishak, Abeba Birhane
August 10, 2026
1 citations
Key Takeaways
-
Most AI alignment research lacks clear definitions of "values": The study found that the majority of 94 analyzed papers do not explicitly define what they mean by human values, instead relying on "preferences" as a substitute—a approach that risks oversimplifying complex, culturally-specific concepts into simple binary choices.
-
Current methods may narrow whose values get represented: As the field shifts away from human annotators toward synthetic data and automated rating systems for training and evaluating AI models, there is a risk of closing off alternative methods for different groups to contest and shape what values are actually built into foundation models.
-
The field needs more philosophical clarity: The paper argues that AI alignment researchers should make their underlying philosophical assumptions about values explicit, rather than leaving them implicit, to enable more informed debate about whether AI systems can truly be aligned with human values.
-
"Preference" is not an adequate substitute for "values": The heavy reliance on preferences in alignment work is problematic because it reduces culturally situated and complex human values—which involve ethics, meaning, and social context—down to measurable preference data.
Abstract
Can AI systems be aligned to human values? The popularization of large language models (LLMs) and multi-modal foundation models has seen a rise in harms spanning from toxic speech and hallucinations to AI agents executing unauthorized actions. Within the field of AI safety, these harmful instances are often framed as the alignment problem, or of models being misaligned with human values. Researchers have responded by pursuing applied and theoretical AI value alignment efforts, often without specifying what they mean by human values. How does the field of AI value alignment conceive of human values? How are these conceptions of values technically operationalized and evaluated? What does the emergent theory of value from this field signify for the future of AI? We annotated 94 value alignment research papers to discern their implicit theory of values in AI. The majority do not define values, relying heavily on preferences as a stand in that runs the risk of reducing complex culturally situated concepts down to binary choices. As researchers dispense with using human annotators for model training and evaluation, turning instead to synthetic data and autorater approaches to aligning and evaluating models, we identify the potential to close off alternative methods for contesting and enacting values in foundation models. In making AI value alignments philosophical commitments explicit, we seek to bring great specificity and under explored perspectives in the debate on whether and how AI can address human values.