Surfacing Semantic Orthogonality Across Model Safety Benchmarks: A Multi-Dimensional Analysis

Safety benchmarks measure different types of harm: The study analyzed five major AI safety datasets and found they focus on distinctly different categories of potential harms—for example, GretelAI emphasizes privacy concerns while WildGuardMix prioritizes self-harm scenarios—meaning no single benchmark comprehensively covers all safety risks.

Significant gaps exist in current safety testing: The research identified six primary harm categories but found uneven representation across benchmarks, revealing that some important safety concerns may be under-tested depending on which benchmark is used to evaluate an AI system.

How questions are asked affects what gets measured: The analysis discovered major differences in prompt length and structure across benchmarks, suggesting that the way safety questions are formulated can influence what types of harms are actually detected, potentially creating blind spots in AI safety evaluation.

Current safety benchmarks are not interchangeable: With a silhouette score of 0.470, the study quantifies that these benchmarks are semantically distinct enough that using one benchmark alone provides an incomplete picture of an AI system's safety—organizations need multiple complementary tests rather than relying on a single safety dataset.

Better transparency is needed for AI safety claims: By mapping which specific harms each benchmark covers, this research enables clearer communication about what safety testing actually means, helping stakeholders understand both what risks are being checked and which potential harms might still be unexamined.