LLM Robustness Leaderboard v1 --Technical report

All tested LLMs have vulnerabilities: The evaluation found that 37 out of 41 state-of-the-art language models could be successfully attacked using automated red-teaming techniques, demonstrating that current safeguards are not foolproof regardless of model size or developer.

Some models are significantly harder to break than others: While nearly all models were vulnerable, the difficulty of attacking them varied by over 300-fold, meaning some models required substantially more attempts to elicit harmful behavior—suggesting meaningful differences in robustness exist and should inform model selection.

Different attack methods work better for different harms: The research identified which specific jailbreaking techniques are most effective for particular types of harmful content, providing actionable insights for developers to strengthen defenses against the most dangerous attack vectors.

Robustness requires ongoing collaborative testing: The involvement of "trusted third parties from the AI Safety Network" in the evaluation demonstrates that identifying vulnerabilities works best through distributed, community-wide efforts rather than isolated internal testing by individual companies.

Current metrics miss important nuances: Moving beyond simple "success or failure" measurements to track how many attempts are needed reveals that vulnerability severity varies dramatically across models, providing a more realistic picture of real-world robustness than binary metrics alone.