A Grading Rubric for AI Safety Frameworks

Companies are rapidly adopting safety frameworks, but they need independent evaluation: The paper notes that major AI companies like Anthropic, OpenAI, and Google DeepMind have published safety frameworks, with 13 additional companies planning to release them by February 2025. However, without a standardized way to assess these frameworks, it's difficult for governments, academia, and the public to determine which companies are genuinely prioritizing safety.

A new grading system can help compare AI safety efforts: The paper proposes a rubric with seven evaluation criteria and 21 specific indicators that can grade frameworks on a scale from A to F. This creates a common language for assessing whether companies' safety commitments are comprehensive and credible, similar to how other industries use standardized evaluation methods.

Independent audits and expert review are essential for accountability: The paper suggests three methods for applying the rubric—surveys, Delphi studies (structured expert consensus), and audits—emphasizing that evaluation should involve external parties rather than relying solely on companies' self-assessments. This approach mirrors accountability mechanisms used in other high-stakes industries.

The goal is to encourage competition on safety, not just capability: By enabling transparent comparisons between frameworks, the rubric aims to create incentives for companies to improve their safety practices. This "race to the top" approach could make responsible AI development a competitive advantage rather than a compliance burden.