Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback

Diverse human feedback creates a real aggregation problem: When training AI systems like GPT-4, different humans often have conflicting preferences about what the model should do. The paper argues this isn't a minor technical issue but a fundamental challenge that requires systematic approaches to combine divergent viewpoints into coherent model behavior.

Social choice theory offers proven frameworks: Rather than inventing new methods, the authors propose applying social choice theory—the academic field that studies how to aggregate individual preferences into collective decisions—to AI alignment. This field has centuries of research on handling exactly this type of preference aggregation problem.

Current approaches lack principled aggregation methods: Existing fine-tuning methods like reinforcement learning from human feedback and constitutional AI don't have explicit, transparent ways of handling disagreement among humans. The paper suggests this gap leaves important decisions about model behavior to ad-hoc engineering choices rather than principled frameworks.

The stakes are high for fairness and representation: How AI systems aggregate human feedback directly affects whose values get represented in model behavior. Using social choice approaches could help ensure that minority viewpoints aren't systematically excluded and that the aggregation process itself is transparent and defensible.

This is an emerging research agenda: The paper documents a December 2023 workshop bringing together researchers to explore this intersection, suggesting this is an active but still-developing area where social choice and AI alignment communities are beginning to collaborate.