AISafetyBenchExplorer: A Metric-Aware Catalogue of AI Safety Benchmarks Reveals Fragmented Measurement and Weak Benchmark Governance

The AI safety benchmark landscape is fragmented rather than scarce: While 195 benchmarks exist for evaluating LLM safety, they lack standardization in how they measure and define safety, creating a "fragmented measurement ecosystem" rather than a coherent one.

Identical metric names hide fundamentally different measurements: Labels like "accuracy," "F1 score," and "safety score" appear across benchmarks but often use "materially different judges, aggregation rules, and threat models," making it difficult to compare results across studies.

Most benchmarks are not actively maintained or widely adopted: Only 7 of the 195 benchmarks reach "Popular" tier status, while 137 have stale GitHub repositories and 96 have outdated Hugging Face datasets, suggesting many resources become abandoned after publication.

Safety evaluation is heavily concentrated in English and lacks diversity: 165 of 195 benchmarks evaluate only English-language models, and 170 are evaluation-only resources without accompanying tools or implementations, limiting accessibility and reproducibility.

Weak governance and publication practices undermine benchmark reliability: The field relies heavily on arXiv preprints rather than peer-reviewed venues for benchmark papers, and lacks standardized metadata recording, making it difficult to track benchmark provenance and ensure accountability.