SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Models

Most tested LLMs have significant safety gaps: The study found that while some models refused all unsafe requests, most gave unsafe responses to over 20% of test prompts, with some reaching over 50% unsafe responses across 100 test prompts spanning five harm categories.

Safety prompts help but aren't sufficient: Adding a safety-emphasizing system prompt substantially reduced unsafe responses, but did not eliminate them entirely, suggesting that additional safeguards beyond instruction-based approaches are needed.

Automated safety filters vary widely in effectiveness: The researchers evaluated five AI safety filters and found considerable performance differences, with the widely-used Perspective API achieving 72% accuracy while a GPT-4 based approach reached 89% accuracy, indicating that current automated safety evaluation tools are inconsistent.

Systematic testing is feasible and necessary: SimpleSafetyTests demonstrates that rapid, systematic identification of safety risks is possible using just 100 carefully designed test prompts, providing a practical methodology for organizations to evaluate their models before deployment.

Safety performance differs across harm categories: The study found differences in how well models performed across the five harm areas tested, suggesting that some types of harmful content may be easier or harder for LLMs to refuse, which has implications for targeted safety improvements.