Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability

VLMs can "cheat" by relying too heavily on text: Vision-Language Models often answer visual questions using only the text information in datasets, rather than truly combining visual and textual understanding. This cross-modal redundancy means current evaluation methods may incorrectly assess how well these systems actually use images.

Current explainability tools give misleading results: Existing methods for explaining VLM decisions rank visual and textual contributions in contradictory ways (with near-zero correlation), indicating that popular explainability approaches fail to accurately measure how vision and language work together in these systems.

A new evaluation metric reveals hidden problems: The paper introduces "Synergistic Faithfulness," which properly measures when vision and language genuinely interact. Testing this metric across multiple VLM systems shows that widely-used explainers over-emphasize visual appearance while missing true cross-modal reasoning.

Better explainability is critical for high-stakes applications: Because VLMs are increasingly deployed in important domains (healthcare, autonomous systems, etc.), having reliable ways to audit their reasoning—rather than just their outputs—is essential for responsible AI development and safety.

Attention-based methods outperform newer approaches: Simpler, adapted attention-based explainers better capture how vision and language actually combine in VLMs compared to newer methods specifically designed for these systems, suggesting that explainability research may have moved in the wrong direction.