Bridging the Missing-Modality Gap: Improving Text-Only Calibration of Vision Language Models

Vision-language models become unreliable when used with text alone: The paper demonstrates that removing images from vision-language models causes "large drops in accuracy and severe miscalibration," meaning the models not only perform worse but also become overconfident in incorrect answers—a critical safety concern for deployed AI systems.

Missing images can't be solved by better text descriptions alone: Even when text provides detailed descriptions containing key information, the models still suffer from unreliable confidence levels. This reveals a fundamental limitation: the models have learned to depend on visual input in ways that text alone cannot replicate, regardless of description quality.

Lightweight solutions can restore reliability without retraining: The proposed Latent Imagination Module (LIM) improves both accuracy and calibration by predicting what visual information would look like without actually generating images. This offers a practical, computationally efficient fix for organizations already using vision-language models in text-only settings.

The fix works across different scenarios and tasks: LIM's improvements hold across "text-only benchmarks, unseen tasks, and missing-image scenarios," suggesting the approach is robust and generalizable rather than a narrow fix for specific situations.

Responsible deployment requires addressing hidden model failures: This research highlights that deploying AI systems outside their training conditions (images + text vs. text-only) can create hidden reliability problems that aren't obvious from surface-level performance metrics alone.