How Should AI Safety Benchmarks Benchmark Safety?

Current safety benchmarks have fundamental flaws: The paper reviewed 210 AI safety benchmarks and found they suffer from "significant technical, epistemic, and sociotechnical shortcomings," meaning they often fail to accurately measure what they claim to measure or don't capture real-world safety concerns.

We need to borrow from established safety practices: The authors recommend that AI safety benchmarking adopt principles from engineering sciences and traditional risk management fields that have decades of proven experience, rather than inventing entirely new approaches—this could substantially improve how well benchmarks actually predict safe AI behavior.

Many safety metrics lack scientific rigor: The paper emphasizes the need for "robust probabilistic metrics" and proper "measurement theory," suggesting that current benchmarks often use imprecise or oversimplified ways of measuring safety that don't account for uncertainty or real-world complexity.

We can't measure everything that matters: The research maps "the space of what can(not) be measured," acknowledging that some important safety concerns may be difficult or impossible to capture in benchmarks—a crucial reality check for practitioners who might over-rely on test scores.

Practical guidance is now available: The authors provide "workflow-oriented guiding questions" and concrete recommendations to help researchers and companies build better safety benchmarks, making this research actionable for organizations deploying AI systems.