VC-Soup: Value-Consistency Guided Multi-Value Alignment for Large Language Models

The core problem: Current methods for aligning LLMs with multiple human values are expensive (requiring separate models for each value combination) and struggle when values conflict with each other, making it difficult to serve diverse user needs fairly.

A practical solution: VC-Soup filters out training data that creates conflicts between values and then combines multiple aligned models together, rather than training entirely new models for each value combination—potentially reducing computational costs significantly.

Value consistency matters: The researchers developed a metric to identify which preference pairs in training data are "value-consistent" (coherent across multiple values), finding that removing inconsistent data helps models better balance competing human values rather than sacrificing performance on some values to excel at others.

Real-world relevance: As LLMs increasingly make decisions and generate content across the internet, the ability to align them with multiple human values simultaneously—rather than optimizing for just one—is essential for trustworthy AI that serves diverse populations and perspectives.

Pareto optimization approach: The final method uses "Pareto filtering" to identify solutions that represent genuine trade-offs between values, meaning improvements in one value don't come at the expense of others—helping stakeholders understand what's actually achievable rather than false compromises.