ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Models

Automated testing reveals significant safety gaps: Despite built-in safeguards, state-of-the-art language models show performance drops of up to 11% when presented with semantically similar prompts asking the same question in different ways, indicating that safety protections are inconsistent and unreliable.

Adversarial prompts expose critical vulnerabilities: The research found error rates as high as 19% when models face adversarial prompts designed to bypass safety measures, which raises serious concerns about physical safety risks when these systems are deployed in real-world applications.

Safety performance varies significantly by domain: The study partitioned safety tests into four different domains and found that model performance differs substantially across these areas, suggesting that a one-size-fits-all safety evaluation approach is insufficient and organizations need domain-specific testing.

Systematic red teaming is essential for responsible deployment: The ASSERT framework demonstrates that comprehensive, automated safety testing using multiple methods (semantic variation, scenario expansion, and adversarial techniques) is necessary to identify hidden vulnerabilities before deployment, rather than relying solely on existing safeguards.

Current evaluation methods are inadequate: The paper's findings suggest that existing safety measures in deployed models are not robust enough to handle the variety of real-world prompting scenarios users will encounter, highlighting the need for more rigorous pre-deployment testing standards.