On the Value of Labeled Data and Symbolic Methods for Hidden Neuron Activation Analysis

Symbolic AI methods can help demystify neural networks: The paper demonstrates that combining symbolic reasoning (OWL-based logic) with large knowledge bases (2 million Wikipedia concepts) can automatically generate human-readable explanations for what individual neurons in deep learning systems detect, moving beyond vague activation patterns toward interpretable concepts.

Background knowledge matters for AI explainability: Using a structured concept hierarchy as reference material significantly improved the quality of neuron interpretations, suggesting that responsible AI systems benefit from being grounded in organized, factual knowledge rather than relying solely on statistical patterns learned from data.

Labeled data and systematic verification are underexplored: The paper identifies that while some hidden neuron activations can be interpretable, there is a significant gap in automated methods that can both hypothesize and verify what neurons are actually detecting—a critical capability for trustworthy AI systems that goes beyond current state-of-the-art approaches.

Model-agnostic approaches offer practical advantages: The proposed method works independently of how the neural network was built, meaning organizations could potentially apply this explanation technique to existing AI systems without rebuilding them, making responsible AI practices more feasible to implement.