Asymptotic Universal Alignment: A New Alignment Framework via Test-Time Scaling

Test-time scaling offers a practical path to serving diverse user preferences: Rather than trying to create a single "perfect" response, the paper proposes having AI models generate multiple candidate responses (k≥1) and letting users choose their preferred option. This approach could help address the fundamental challenge that different users have conflicting values and preferences.

Current alignment methods may be leaving performance gains on the table: Popular techniques like Nash learning from human feedback (NLHF), while optimal for single responses, fail to improve meaningfully when generating multiple options. The paper shows these methods can collapse to producing the same response repeatedly, making additional samples "redundant"—suggesting current approaches aren't fully leveraging the benefits of generating multiple outputs.

Optimal diversity-preserving methods can achieve predictable improvement rates: The paper demonstrates that properly designed alignment methods can achieve a win rate of k/(k+1) against single-output competitors, meaning performance improves reliably as more options are generated. For example, with 9 candidate responses, a user could expect roughly 90% win rates compared to any single-output model.

Output diversity is critical for responsible AI alignment: The core insight is that maintaining diverse responses—rather than converging to one majority-preferred answer—is essential for both serving heterogeneous preferences and enabling meaningful test-time scaling benefits. This suggests future alignment work should prioritize preserving response variety.