Explaining Vision GNNs: A Semantic and Visual Analysis of Graph-based Image Classification

GNNs for image classification organize images into patch-based graphs rather than processing raw pixels, offering a potentially more interpretable alternative to traditional deep learning approaches, though the paper finds that explaining how these models actually work has been largely neglected despite graphs' theoretical interpretability advantages.

The models' internal reasoning diverges from human perception, particularly in deeper layers, meaning that even though we can visualize what the model is focusing on, the semantic relationships it learns don't necessarily match how humans would naturally understand images—raising questions about whether these models are learning robust or trustworthy features.

Graph connections within these models don't consistently preserve meaningful object structures and spatial relationships across different layers, suggesting that the patch-based graphs may not maintain semantic coherence throughout the decision-making process, which could be a concern for reliability in high-stakes applications.

Adversarial robustness testing reveals differences in model explanations under attack conditions, indicating that the visual explanations we can generate may not fully reflect whether the model's reasoning is stable or vulnerable to manipulation—an important consideration for deploying these systems responsibly.

Heatmap visualizations can effectively show where models focus attention, but this surface-level explainability may mask deeper issues with how the models actually reason, highlighting the gap between "explainable" visualizations and genuine understanding of model decision-making.