Policy-Grounded Safety Evaluation of 20 Large Language Models

Safety performance varies dramatically across models and domains: The study found mean safety scores ranging from 86.2% to 52.4% across 20 commercial LLMs, indicating that some models are substantially safer than others depending on the specific safety domain being tested.

Models excel at preventing misinformation but struggle with privacy protection: While LLMs achieved a mean score of 95.7% in preventing misinformation, they performed poorly in Privacy & Impersonation scenarios with a mean score of only 24.3%, revealing critical gaps in protecting personal information.

Safety is context-dependent and inconsistent: The research demonstrates that LLM safety is not uniform—models that perform well in one domain may fail in another. This suggests that blanket safety claims about any single model are misleading without domain-specific evaluation.

Scalable, policy-based evaluation tools are necessary for oversight: The paper introduces Aymara AI as a programmatic platform that can automatically generate safety tests based on real-world policies and rate model responses, offering a practical solution for ongoing safety monitoring as these systems become more widely deployed.

Current safety evaluations are insufficient for real-world deployment: The significant performance disparities and failures in complex domains suggest that existing safety measures are inadequate, and organizations deploying LLMs need rigorous, customized evaluation frameworks to identify and mitigate risks specific to their use cases.