Gradient-Based Interpretability Methods and Binarized Neural Networks

Interpretability methods don't work equally well across different neural network types. The paper tested three popular explanation techniques (Gradient, SmoothGrad, and GradCAM) on two types of networks and found significant differences in how well they performed, suggesting that researchers cannot assume an interpretability method that works for standard networks will work for specialized variants.

SmoothGrad produces unreliable explanations for Binarized Neural Networks. When applied to BNNs, SmoothGrad generated "significantly noisier maps" compared to full precision networks, meaning the visual explanations of what the network was focusing on became harder to interpret and trust.

GradCAM sometimes produces nonsensical explanations for BNNs. The paper found that some Binarized Neural Networks generated saliency maps using GradCAM that appeared meaningless, raising concerns about whether users could actually trust these explanations to understand the network's decision-making.

Binarized Neural Networks need their own interpretability validation. Since BNNs are increasingly used in edge computing (like on mobile devices and IoT systems), it's critical that interpretability techniques be specifically tested and validated for these networks rather than assuming methods developed for standard networks will transfer over.

Interpretability research should test across diverse network architectures. The authors argue this work demonstrates a broader need for the AI community to evaluate explanation methods on a wider range of network types, not just the most common ones.