Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)

Language models show a "Artificial Hivemind effect" where different models produce strikingly similar outputs to open-ended questions, not just within a single model but across different models—raising concerns that widespread AI use could homogenize human thought and creativity through repeated exposure to similar AI-generated content.

Current methods for evaluating AI diversity are severely limited, typically only measuring narrow tasks like random number generation rather than real-world creative scenarios, which is why the researchers created Infinity-Chat, a dataset of 26,000 diverse real-world queries to enable more comprehensive diversity assessment.

The homogeneity problem is particularly pronounced for open-ended creative tasks like brainstorming and ideation—the very areas where human diversity of thought is most valuable—suggesting that relying on current language models for creative work may inadvertently narrow rather than expand human possibilities.

Human preferences for AI outputs are more diverse and nuanced than AI systems recognize, as evidenced by 31,250 human annotations showing that language models, reward models, and AI judges are "less" aligned with the full spectrum of individual human preferences, meaning AI systems may be optimizing for a narrow slice of what humans actually want.

We need better tools and datasets to measure AI homogeneity at scale, particularly for open-ended tasks—this work provides the first comprehensive taxonomy of open-ended prompts (6 categories, 17 subcategories) that can help researchers and developers identify and address diversity problems before deploying AI systems widely.