Pluralis v0.1: Towards a Multicultural, Multimodal, Multilingual Benchmark for AI Risk and Reliability

Western bias in AI safety is a blind spot: Current AI safety benchmarks rely on "Western-centric culture-agnostic defaults" that fail to account for regional laws, cultural taboos, and social norms, meaning AI systems deployed globally are vulnerable to causing harm in ways their creators never tested for.

Context matters more than individual elements: The paper demonstrates that seemingly harmless combinations—like asking about gifting a clock (which is taboo in some cultures) with an image of that clock—can trigger cultural or legal violations that AI systems miss when evaluating text or images separately, highlighting the importance of multimodal evaluation.

AI failures are locale-specific and systematic: Vision-Language Models show recurring, predictable failure patterns that vary by region, including misidentifying images, missing cultural context, and failing to refuse requests appropriately—suggesting these aren't random errors but systematic gaps in how models understand different cultural contexts.

Cultural appropriateness deserves equal weight to universal safety: The research establishes that evaluating whether AI respects local cultural norms should be treated as a primary safety concern alongside universal harms, not as an afterthought, requiring dedicated datasets and evaluation methods for each region.

Responsible global AI deployment requires local expertise: The Pluralis dataset was built by sourcing hazards natively from six Asia-Pacific countries across eight languages, suggesting that truly safe AI systems require input from people embedded in local cultures rather than adapted Western frameworks.