Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging

Model merging can be more effective than mixing training data for creating AI systems that are simultaneously helpful, honest, and harmless. The paper found that merging specialized models improved performance by 1-3% compared to existing methods, and 2-5% compared to data mixture approaches, suggesting this parameter-level approach deserves more attention in responsible AI development.

The "3H" goals of AI systems sometimes conflict with each other, creating difficult trade-offs. The research reveals previously overlooked relationships showing that optimizing for helpfulness, honesty, and harmlessness simultaneously involves navigating genuine tensions between these objectives—not just combining them independently.

Current data mixture methods have significant limitations that make them less reliable for balanced AI alignment. These approaches depend heavily on expert knowledge to decide how to combine training data and struggle with conflicting optimization signals, making them less robust than parameter-level alternatives.

A new technique called RESM addresses practical challenges in model merging by using smarter weighting strategies and adapting to different layers of neural networks. This method tackles real problems that arise when combining multiple specialized models, making the merging approach more practical for real-world deployment.

Achieving responsible AI requires moving beyond simple training data adjustments to more sophisticated technical approaches. The findings suggest that responsible AI development benefits from exploring parameter-level solutions that can better handle the complex, sometimes conflicting demands of creating helpful, honest, and harmless systems.