ThaiSafetyBench: Assessing Language Model Safety in Thai Cultural Contexts

Safety testing has a language gap: Most AI safety evaluations focus on English, leaving speakers of other languages like Thai vulnerable to undetected risks. This research reveals that language models may have different safety vulnerabilities depending on the language they're used in.

Cultural context matters for AI safety: The study found that attacks specifically designed around Thai cultural, social, and contextual nuances were significantly more successful than general attacks in Thai. This suggests that safety measures need to account for local values and contexts, not just translate English-based safety standards.

Open-source models pose greater risks than closed-source ones: The evaluation of 24 language models showed that closed-source models (like GPT-4 and Gemini) generally performed better on safety measures than openly available models, raising concerns about the robustness of freely available AI tools that anyone can deploy.

Current AI safety methods have blind spots: The research demonstrates that existing safety alignment techniques—the methods used to make AI systems behave responsibly—don't adequately protect against culturally-informed attacks, suggesting that safety approaches need fundamental improvements beyond current practices.

Reproducible safety evaluation tools are now available: The researchers released ThaiSafetyBench (1,954 test prompts), a classifier tool, and a public leaderboard to enable ongoing safety monitoring. This provides a model for how other languages and cultures can systematically evaluate and improve AI safety.