Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguards

Multi-modal AI systems face fundamentally new safety risks that differ from single-mode AI: when language models are combined with image and video understanding, the interaction between these different types of information creates novel vulnerabilities—such as "modality misalignment"—that existing safety tools designed for text-only systems cannot adequately address.

Adversarial attacks have evolved to exploit cross-modal weaknesses: attackers can now manipulate images or other non-text inputs in ways that fool the combined system, even when the text safeguards remain intact, representing a significant gap in current safety frameworks that primarily focus on text-based threats.

Hallucinations and false outputs are more complex in multi-modal systems: when language models combine information from multiple sources (text, images, video), they can generate convincing but false information in new ways, making it harder to detect and prevent misinformation compared to text-only models.

Current safety solutions are insufficient for multi-modal AI: the survey emphasizes that safety mechanisms built for traditional language models don't automatically transfer to systems that integrate multiple types of information, requiring fundamentally new approaches to safety design and testing.

The field lacks standardized safety assumptions for multi-modal AI: unlike established safety practices for single-mode systems, there is no consensus framework for how to safely build, test, and deploy multi-modal AI systems, leaving developers without clear guidelines for responsible development.