Intersectionality in Conversational AI Safety: How Bayesian Multilevel Models Help Understand Diverse Perceptions of Safety

Safety perception is not universal: The study of 252 raters evaluating 1,340 conversations found that demographic characteristics—including race/ethnicity, gender, and age—significantly influence whether people consider the same chatbot conversation safe or unsafe, meaning a one-size-fits-all safety standard may not adequately protect all users.

Intersectionality matters more than single factors: The research demonstrates that intersectional effects (where multiple demographic characteristics combine) are particularly important, especially the intersection of race/ethnicity and gender, suggesting that safety risks cannot be understood by examining demographic groups in isolation.

Current safety evaluations may have blind spots: Because safety labeling often relies on raters from limited demographic backgrounds, the study implies that existing chatbot safety assessments may miss harms that are particularly salient to underrepresented groups, potentially leaving vulnerable populations at greater risk.

Diverse perspectives are essential for responsible AI development: The use of a stratified rater pool across gender, age, race/ethnicity, and locale demonstrates that including demographically diverse human evaluators in safety testing is a practical, necessary step for identifying and mitigating AI harms that affect different communities differently.

Technical methods can illuminate social complexity: Bayesian multilevel modeling provided a rigorous way to untangle how multiple demographic factors interact to shape safety perceptions, offering a methodological approach that other AI developers could adopt to better understand whose voices are being heard in safety assessments.