SafetyBiasFairnesscs.CL
SEA-SafeguardBench: Evaluating AI Safety in SEA Languages and Cultures
Panuthep Tasawong, Jian Gang Ngui, Alham Fikri Aji, Trevor Cohn, Peerat Limkonchotiwat
December 5, 2025
arXiv.org
Panuthep Tasawong, Jian Gang Ngui, Alham Fikri Aji, Trevor Cohn, Peerat Limkonchotiwat
AI safety systems fail in Southeast Asian languages: State-of-the-art LLMs and guardrails "underperform when compared to English texts" on SEA-SafeguardBench, revealing that safety protections cannot be assumed to work equally across languages and cultures.
Machine translation is not a shortcut for safety evaluation: The paper demonstrates that existing multilingual safety benchmarks that rely on machine-translated English data "fail to capture nuances in low-resource languages," meaning AI safety testing cannot simply translate English benchmarks and expect reliable results.
Cultural context matters for defining harm: SEA languages have "unique safety concerns, from culturally sensitive political speech to region-specific misinformation" that differ from English-speaking contexts, indicating that what counts as harmful content depends on local norms and cannot be universally standardized.
Native expertise is essential for responsible AI: The benchmark required "human-verified" and "natively authored" content across eight languages and 21,640 samples, highlighting that responsible AI development demands local expertise rather than top-down approaches from English-speaking AI centers.
Linguistic diversity is a blind spot in current AI safety: Southeast Asian languages remain "underrepresented" in safety research despite the region's "linguistic diversity," suggesting that billions of people using these languages may be using AI systems with inadequate safety protections.
Safeguard models help large language models (LLMs) detect and block harmful content, but most evaluations remain English-centric and overlook linguistic and cultural diversity. Existing multilingual safety benchmarks often rely on machine-translated English data, which fails to capture nuances in low-resource languages. Southeast Asian (SEA) languages are underrepresented despite the region's linguistic diversity and unique safety concerns, from culturally sensitive political speech to region-specific misinformation. Addressing these gaps requires benchmarks that are natively authored to reflect local norms and harm scenarios. We introduce SEA-SafeguardBench, the first human-verified safety benchmark for SEA, covering eight languages, 21,640 samples, across three subsets: general, in-the-wild, and content generation. The experimental results from our benchmark demonstrate that even state-of-the-art LLMs and guardrails are challenged by SEA cultural and harm scenarios and underperform when compared to English texts.
Under review
Citation data from Semantic Scholar