How Are LLMs Mitigating Stereotyping Harms? Learning from Search Engine Studies

Safety training focuses on legal risks over social harms: The paper finds that LLM developers have prioritized "safety" training to address legal liabilities rather than conducting thorough social impact evaluations, mirroring earlier patterns observed with search engine development.

Significant gaps remain in addressing stereotypes about marginalized groups: Despite improvements with safety prompts, LLMs still produce toxic stereotyping outputs at concerning rates, particularly for prompts about "peoples/ethnicities and sexual orientation," indicating these harms receive insufficient attention during model development.

Intersectional identities face disproportionate stereotyping: The research reveals that mentions of intersectional identities (people with multiple marginalized characteristics) trigger significantly more stereotyping than single-identity prompts, suggesting current mitigation approaches inadequately address compounded discrimination.

Safety prompts help but aren't sufficient: While system prompts do improve stereotyping outputs, the paper demonstrates this alone is not a complete solution, indicating the need for more comprehensive approaches to harm mitigation during training and evaluation phases.

Urgent need for accountability across the AI pipeline: The authors call for greater responsibility from model builders, academics, and policymakers regarding training data curation, evaluation metrics design, and mandatory social impact measurement to prevent stereotyping harms—especially as LLMs become integrated into search engines.