AI Companies Should Report Pre- and Post-Mitigation Safety Evaluations

Current safety reports paint an incomplete picture: The paper finds that AI companies rarely evaluate and report both the raw capabilities of their models (pre-mitigation) and how safe those models become after safety measures are applied (post-mitigation). Looking at only one stage can mislead policymakers about whether a system is actually safe to deploy.

Lack of standardization makes comparison impossible: The paper identifies that different AI companies use different evaluation methods with varying rigor, making it difficult for regulators to compare safety across systems or set consistent safety standards across the industry.

Vague reporting prevents effective oversight: The analysis shows that companies often report safety results in ways that are too unclear or general to help policymakers make informed decisions about deployment, access restrictions, or safety requirements.

Transparency should be tiered, not one-size-fits-all: The paper recommends that detailed pre- and post-mitigation safety data be disclosed to approved government bodies (not necessarily the public), while companies maintain minimum transparency standards for public reporting—balancing the need for regulatory oversight with legitimate confidentiality concerns.

Standardized evaluation methods are essential: The paper calls for the development of common evaluation approaches across the industry, which would allow regulators to meaningfully assess and compare the safety of different AI systems and hold companies accountable to consistent standards.