Evaluating AI Providers' Frontier Safety Frameworks

Current safety frameworks lack concrete accountability: The 12 AI company frameworks score between 8-34% on established risk management criteria, with a median of just 18%, meaning most commitments are vague enough that it's difficult to verify whether companies are actually following through on their promises.

Frameworks are too underspecified to guide decisions: The assessment found that many aspects of the frameworks are missing or under-specified, making it nearly impossible for regulators, policymakers, or the public to predict how companies will actually respond to safety risks or determine if their planned responses are adequate.

Better practices already exist within the industry: A company that adopted all the leading practices already being used by at least one of the 12 companies assessed could score 54%—triple the current median—suggesting that significantly stronger commitments are feasible without waiting for new technology or expertise.

Legislation is treating incomplete frameworks as accountability mechanisms: Following the 2024 AI Seoul Summit, emerging laws are incorporating these frameworks into official reporting requirements, despite their current limitations as accountability tools.

The low scores may reflect industry immaturity rather than unwillingness: The researchers note that low scores are "natural given the nascency of AI risk management compared to industries with decades of practice" like aviation and nuclear power, suggesting the frameworks' weakness may stem from the field's newness rather than bad faith.