Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?

Many AI safety benchmarks may not measure what they claim to measure. The paper finds that popular safety benchmarks correlate strongly with general AI capabilities and training compute rather than measuring safety improvements independently, meaning a model could appear "safer" simply because it's more capable overall.

"Safetywashing" is a real risk in the field. Companies and researchers could misrepresent ordinary capability improvements as genuine safety progress, potentially misleading stakeholders about actual risk mitigation—similar to how "greenwashing" misrepresents environmental efforts.

The AI safety field lacks clear definitions and consistent measurement standards. Without agreed-upon definitions of what "AI safety" actually means in technical terms, it's difficult for researchers to know if they're making real progress or just creating benchmarks that conflate safety with general intelligence.

Current safety metrics need fundamental redesign. The paper calls for safety measurements that are "empirically separable from generic capabilities advancements," meaning we need new evaluation methods that can distinguish genuine safety improvements from improvements that simply come from making AI systems more powerful.

Rigorous evaluation frameworks are essential for responsible AI development. By establishing clearer definitions and better metrics for safety, the research community can provide more honest assessments of AI risks and ensure that safety research actually addresses emerging dangers rather than creating an illusion of progress.