VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety

Joint multimodal reasoning reveals critical safety gaps: While state-of-the-art models achieve over 90% accuracy on safety tasks involving single inputs (image or text alone), performance drops dramatically to 20-55% when models must reason about how images and text combine to create harmful content—a scenario that existing safety evaluations often miss.

One-third of safety failures occur despite correct individual analysis: The research found that 34% of errors in joint image-text safety classification happen even when models correctly identify risks in the image and text separately, indicating models lack the compositional reasoning needed to understand how benign elements become harmful together.

Current safety systems create a problematic trade-off: Models struggle to distinguish genuinely unsafe content from borderline cases that warrant engagement, leading to either over-blocking of legitimate requests or under-refusal of actual harms—a balance problem that affects user experience and safety simultaneously.

Comprehensive evaluation framework identifies 15 distinct harm categories: The VLSU benchmark includes 8,187 human-annotated samples across 15 harm categories with fine-grained severity classifications, providing a more nuanced foundation for safety testing than previous approaches that treated vision and language separately.

Safety evaluation methodology needs fundamental redesign: The findings demonstrate that evaluating multimodal models' safety requires systematic testing of how different input combinations interact, not just assessing individual modalities in isolation.