ExplainabilityTransparencyAccountabilitycs.LG
Explaining How Deep Neural Networks Forget by Deep Visualization
Giang Nguyen, Shuan Chen, Tae Joon Jun, Daeyoung Kim
May 3, 2020
ICPR Workshops
11 citations
Key Takeaways
-
Neural networks can "forget" previous knowledge when learning new tasks, a problem called catastrophic forgetting. This paper's visualization tool (CFD) reveals exactly which parts of the network are responsible for this forgetting, making the previously opaque process more understandable.
-
Interpretability and performance can work together: The paper demonstrates that understanding how networks forget (through visualization) led to a better solution called Critical Freezing that outperforms existing methods, suggesting that explainability isn't just about transparency—it can improve AI system quality.
-
Specific network components are responsible for forgetting: By examining ResNet architecture, the researchers identified which layers and components lose information about previous tasks, moving beyond vague explanations to pinpoint exactly where problems occur in the network's structure.
-
Better explanations of AI behavior are essential for responsible deployment: The authors emphasize that as deep neural networks are "adopted over diverse aspects of human life," understanding their behaviors is "critical"—suggesting that explainability tools like CFD are necessary safeguards for trustworthy AI systems in real-world applications.
Abstract
Explaining the behaviors of deep neural networks, usually considered as black boxes, is critical especially when they are now being adopted over diverse aspects of human life. Taking the advantages of interpretable machine learning (interpretable ML), this paper proposes a novel tool called Catastrophic Forgetting Dissector (or CFD) to explain catastrophic forgetting in continual learning settings. We also introduce a new method called Critical Freezing based on the observations of our tool. Experiments on ResNet articulate how catastrophic forgetting happens, particularly showing which components of this famous network are forgetting. Our new continual learning algorithm defeats various recent techniques by a significant margin, proving the capability of the investigation. Critical freezing not only attacks catastrophic forgetting but also exposes explainability.