Modal Aphasia: Can Unified Multimodal Models Describe Images From Memory?

Multimodal AI models have a fundamental blind spot: Leading AI systems can visually recognize and reproduce images accurately but struggle to describe those same images in text, even though they were trained on both types of information together. The researchers call this "modal aphasia" and show it's a systematic problem, not just a training glitch.

Safety guardrails can be bypassed through different modalities: If a model is trained to refuse harmful requests in text, it may still generate unsafe images on the same topic. This means current AI safety measures that focus on one form of communication (like text filters) leave dangerous vulnerabilities in other forms (like image generation).

The problem appears across different AI architectures: The researchers tested this phenomenon on multiple types of multimodal models and found the same pattern consistently, suggesting this is a core limitation of how these systems currently work rather than an isolated issue with one model.

Visual memory doesn't translate to language: Models can create near-perfect reproductions of complex images (like iconic movie artwork) but fail to accurately describe what they just reproduced in words, revealing a disconnect in how these systems process and communicate information across modalities.

This gap poses real-world risks: The disconnect between visual and textual capabilities creates practical safety concerns for deployed AI systems, as harmful content could evade text-based safeguards by being expressed through images instead.