A Method for Learning Value Systems in Generative AI

Value systems in AI need explicit representation: The paper addresses a critical gap in AI alignment by proposing that generative AI systems should learn explicit, computational representations of human values rather than simply mimicking preferences. This distinction matters because it allows AI systems to understand why certain decisions align with human values, not just what humans prefer.

Learning values from human behavior is feasible but requires structure: The method learns value systems by observing which responses humans prefer in different scenarios (pairwise comparisons). Importantly, it does this while maintaining awareness that human values are multidimensional—meaning it recognizes that people care about multiple, sometimes competing values simultaneously, rather than treating all preferences as interchangeable.

Explainability improves when values are grounded: A key finding is that the approach improves explainability compared to existing methods. This means stakeholders can better understand which values an AI system is prioritizing and how it's weighing them against each other—crucial for trust and accountability in high-stakes applications.

Trade-offs between values can be made transparent: By representing values as a weighted system (like a recipe with specific ingredients and proportions), the method makes visible how different human values are being balanced. This allows organizations and users to see whether an AI system is prioritizing, for example, helpfulness over safety, or accuracy over privacy.

This approach bridges preference learning and value alignment: Rather than treating AI alignment as simply "do what users want," this work positions it as learning coherent value systems that can generalize beyond specific observed preferences to new situations.