Automated Analysis of Global AI Safety Initiatives: A Taxonomy-Driven LLM Approach

LLM choice significantly impacts policy analysis results: The study found that different large language models produced substantially different outcomes when analyzing the same AI safety policy documents, suggesting that automated policy comparisons should not rely on a single model without validation.

Human expertise remains essential for policy evaluation: While three expert human evaluators showed high agreement with each other when comparing policy documents, their judgments still differed from the LLM-generated scores, indicating that AI tools should complement rather than replace expert human review of safety policies.

Automated frameworks can support policy comparison at scale: The crosswalk system successfully extracted and compared activities across policy documents using a shared taxonomy, suggesting that AI-assisted tools could help policymakers and researchers identify similarities and gaps across multiple safety initiatives more efficiently than manual review alone.

Some policy document pairs show high disagreement across models: Certain document comparisons yielded inconsistent similarity scores depending on which LLM was used, highlighting that automated policy analysis works better for some document pairs than others and requires careful interpretation.

Comparative inspection of policies benefits from AI assistance despite limitations: The framework's ability to produce summaries, comparisons, and similarity scores for each policy aspect could help stakeholders understand how different AI safety initiatives align, even though the results should be validated through human expert review.