AIR-BENCH Live: An Evolving Safety Benchmark for Foundation Models

Safety benchmarks need continuous updates: The paper demonstrates that static safety benchmarks quickly become outdated as AI models improve and governments introduce new regulations. AIR-BENCH Live addresses this by automatically monitoring policy changes across seven jurisdictions and updating its risk categories accordingly, expanding from 314 to 335 granular risks in the current version.

Significant safety variation exists across models: When evaluated on the same benchmark, recent foundation models show a wide range of safety performance, with compliance scores ranging from 0.17 to 0.89. This suggests that some models are substantially better at refusing harmful requests than others, highlighting the importance of rigorous safety testing during model development.

Newer safety tests are more challenging: The modernized prompts in AIR-BENCH Live are on average 0.06 points harder than the 2024 version, with the largest performance drops among the most compliant models. This indicates that as jailbreaking techniques evolve, safety benchmarks must also evolve to remain effective at identifying vulnerabilities.

Language creates safety disparities: Most models tested showed modestly lower safety performance on non-English prompts compared to English, suggesting that safety measures may be less robust across different languages—a potential concern for global AI deployment.

Automation enables scalable safety monitoring: By using automated pipelines to track regulations and generate test prompts with minimal human review, AIR-BENCH Live demonstrates a scalable approach to keeping safety benchmarks current with both regulatory changes and emerging attack techniques.