Multimodal Model Diffing for Feature Discovery and Control

Sparse autoencoders can make multimodal AI systems more interpretable: The research successfully decomposed hidden states in three major multimodal models (LLaVA-MORE, PaliGemma 2, and InternVL3.5) into interpretable feature directions, enabling researchers to identify which internal features drive specific behaviors—a critical step toward auditing AI systems.

Researchers can now isolate and control specific AI capabilities: MMDiff's "model diffing" approach identified features that were specifically changed when models gained multimodal abilities, allowing targeted removal or steering of particular behaviors. For example, removing identified features reduced spatial reasoning performance by 12% and OCR accuracy by 17% while leaving general question-answering unaffected.

This approach shows promise for improving AI safety: When applied to multimodal safety attacks, removing causally-identified features reduced attack success rates by 24%, suggesting that feature-level control could help defend against adversarial misuse without degrading legitimate model performance.

Feature-level control is more precise than current alternatives: Unlike existing post-hoc inspection methods, MMDiff enables direct intervention at the feature level, moving beyond simply understanding what models do to actually controlling specific behaviors—a significant advance for responsible AI deployment.

The method works across different model architectures: Successfully applying MMDiff to three different MLLM families suggests this interpretability and control approach could become a generalizable tool for auditing and governing multimodal AI systems.