Beyond Text: Multimodal Jailbreaking of Vision-Language and Audio Models through Perceptually Simple Transformations

Simple visual tricks can bypass AI safety systems: Even though vision-language models show nearly perfect safety performance (0% attack success) when given text-only requests, the same models fail dramatically when images are slightly modified—with attack success rates exceeding 75%. This reveals that AI safety training focuses too heavily on text while neglecting other input types.

Current safeguards don't work across different input types: The research tested attacks across vision, audio, and text modalities and found that models protected against text-based harmful requests remain vulnerable when the same harmful content is presented through images or audio. This "cross-modal gap" suggests safety measures need fundamental redesign to handle multiple input types simultaneously.

Attacks require minimal technical sophistication: Techniques like FigStep-Pro (breaking images into keyword pieces) and simple audio modifications (changing pitch or speed) achieved attack success rates up to 89% on frontier models. The accessibility of these methods—requiring no advanced expertise—means vulnerabilities could be exploited widely if not addressed.

High-risk content categories remain unprotected: The study specifically tested three dangerous categories (harmful content, CBRN materials, and child exploitation material) and found severe vulnerabilities across all of them, indicating that even the most critical safety priorities are inadequately protected in multimodal systems.

Audio inputs present an overlooked vulnerability: Audio-based attacks achieved 25% success rates even for basic technical queries, suggesting that audio modality has received less safety attention than vision or text, creating a potential weak point in multimodal AI systems.