OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMs

Multimodal AI systems have significant safety vulnerabilities: Even leading frontier models like GPT-5.2, Claude 4.5, and Gemini 3 Pro showed average Attack Success Rates as high as 49.14%, meaning nearly half of adversarial attacks successfully bypassed safety measures.

Current safety testing is fragmented and insufficient: Existing red-teaming benchmarks are limited to single-turn text interactions and lack the scalability needed for systematic evaluation, leaving critical gaps in how thoroughly these systems are tested before deployment in important applications.

Advanced reasoning capabilities don't guarantee safety: The paper reveals that reasoning models do not inherently possess superior robustness against complex, multi-turn jailbreak attempts, suggesting that model sophistication alone doesn't solve safety problems.

Standardized, comprehensive testing frameworks are essential: OpenRT's modular approach—integrating 37 diverse attack methodologies across five critical dimensions (model integration, datasets, attack strategies, judging methods, and metrics)—demonstrates the need for unified testing standards to systematically evaluate multimodal AI safety across different attack types.

Generalization across attack types remains a key weakness: Models fail to generalize robustness across different attack paradigms, meaning a system that resists one type of adversarial attack may remain vulnerable to others, highlighting the complexity of achieving comprehensive AI safety.