Disentangling Hallucinations: Orthogonal Semantic Projection for Robust Interpretability

Hallucination in AI explanations is a widespread structural problem, not an isolated bug: The paper demonstrates that when Vision-Language Models like CLIP are asked to explain images with incorrect descriptions (e.g., labeling a dog as a "cat"), their explanation maps still highlight relevant regions. This occurs across multiple architectures, indicating it's a fundamental flaw in how these models work rather than a fixable quirk in any single system.

The root cause is "semantic leakage" in how AI models store information: The researchers identified that high-dimensional embedding spaces—where AI models store meaning—allow unrelated concepts to bleed into each other. This means when a model processes the word "cat," information about dogs and other animals is inadvertently activated, causing explanations to highlight the wrong things even when the model's final answer is correct.

A new geometric solution (OSP) can reduce hallucinations while maintaining accuracy: The proposed Orthogonal Semantic Projection method works by mathematically "blinding" the explanation system to shared features between concepts, forcing it to focus only on what's truly unique to the correct answer. Testing shows this reduces hallucinations while preserving the model's ability to correctly explain accurate predictions.

This matters for safety-critical applications: As Vision-Language Models are deployed in high-stakes domains (medical imaging, autonomous vehicles, security systems), users need to trust that when the AI explains its reasoning, those explanations are actually reliable—not just plausible-sounding stories that happen to highlight the right regions for the wrong reasons.