DICES Dataset: Diversity in Conversational AI Evaluation for Safety

Safety judgments are not universal: The DICES dataset demonstrates that what constitutes "safe" or "unsafe" conversational AI varies significantly based on raters' demographic backgrounds, including race/ethnicity, age, and gender. This challenges the common practice of treating safety as a binary classification problem with clear-cut positive and negative examples.

Current evaluation methods may miss important perspectives: Traditional safety datasets that collapse diverse rater opinions into single labels risk obscuring whose perspectives are being centered in AI safety decisions. By preserving the full distribution of rater votes rather than averaging them, DICES reveals how different demographic groups may have legitimately different safety concerns.

Demographic representation in AI evaluation matters: The paper shows that raters' demographic characteristics intersect with their safety ratings, suggesting that AI safety evaluation teams need diverse representation. Homogeneous evaluation teams may systematically miss safety issues that matter to other communities.

Safety is culturally and socially situated: The research emphasizes that conversational AI safety cannot be treated as a purely technical problem with universal solutions. Safety standards must account for cultural and social context, making it essential to include diverse perspectives in the evaluation process.

Better benchmarks enable better accountability: By providing a shared dataset that captures demographic information and rating variance, DICES allows researchers to measure and compare how different AI models perform across demographic groups, creating opportunities to identify and address disparities in safety outcomes.