ExplainabilityTransparencyTrustworthinesscs.NE
Neural network interpretability with layer-wise relevance propagation: novel techniques for neuron selection and visualization
Deepshikha Bhati, Fnu Neha, Md Amiruzzaman, Angela Guercio, Deepak Kumar Shukla, Ben Ward
December 7, 2024
Computing and Communication Workshop and Conference
15 citations
Key Takeaways
-
Layer-wise Relevance Propagation (LRP) can trace AI decisions back to specific input features, allowing researchers to understand which parts of an image a neural network "looked at" when making decisions—a critical capability for high-stakes applications like medical imaging or autonomous vehicles where accountability matters.
-
Current LRP methods have precision limitations when evaluating individual neuron contributions, meaning existing interpretability tools don't always accurately identify which specific components of a neural network are responsible for particular decisions, highlighting a gap in current transparency techniques.
-
The paper's novel approach improves neuron selection precision using visualization techniques like heatmaps and deconvolutional reconstruction, making it easier to see which internal network components are most important—similar to highlighting the "reasoning steps" an AI system uses rather than treating it as a black box.
-
Better interpretability directly supports trustworthiness in real-world AI applications, particularly in computer vision systems where understanding why an AI made a decision (not just what decision it made) is essential for users, regulators, and affected communities to verify the system is working fairly and correctly.
-
This work demonstrates that improving AI transparency is technically achievable through targeted research, suggesting that the "black box" nature of neural networks is not inevitable but rather a challenge that can be addressed with better tools and methods.