This Looks Like That... Does it? Shortcomings of Latent Space Prototype Interpretability in Deep Networks

Interpretable-by-design models aren't automatically trustworthy: While prototype learning networks like ProtoPNet are marketed as more interpretable alternatives to black-box AI systems, this paper demonstrates they have a critical flaw—the similarity measures they use internally don't reliably match what humans perceive as similar, creating a false sense of transparency.

The "semantic gap" can lead to nonsensical decisions: The researchers found that ProtoPNet can make incomprehensible classifications based on learned prototypes that appear similar in the network's internal mathematical space but look nothing alike to humans, undermining the core promise of interpretability.

Even minor image artifacts can fool interpretable models: The study showed that simple JPEG compression artifacts—common in real-world images—can corrupt the model's decision-making process, suggesting these systems may be more fragile than practitioners assume when deployed in practice.

Intentional adversarial attacks expose deeper vulnerabilities: When researchers deliberately crafted inputs to exploit the latent space gap, they could make the model produce interpretable-looking but fundamentally wrong decisions, raising concerns about potential misuse or unexpected failures in high-stakes applications.

Practitioners need realistic expectations about interpretability: The authors caution that organizations deploying prototype-based models should not assume they've solved the interpretability problem, and should maintain appropriate skepticism about whether the model's stated reasoning actually reflects sound decision-making logic.