Concept backpropagation: An Explainable AI approach for visualising learned concepts in neural network models

Neural networks learn interpretable concepts: The research demonstrates that neural networks don't just process data as meaningless patterns—they actually learn to represent recognizable concepts internally. By using "concept probes," researchers can identify and visualize what specific ideas or features a model has learned to recognize, making the "black box" more transparent.

Visualization reveals what models actually use to make decisions: Concept backpropagation allows researchers to see exactly which parts of input data (images, text, etc.) a model focuses on when representing a particular concept. This helps verify whether models are making decisions based on relevant information or potentially problematic shortcuts—a critical concern for responsible AI deployment.

Entanglement of concepts raises interpretability challenges: The paper notes that concept representations can become "entangled" within neural networks, meaning multiple concepts may be mixed together in how the model processes information. This complexity suggests that even when we identify what concepts a model learns, understanding how they interact remains challenging for ensuring trustworthy AI systems.

Method works across different types of data: The researchers tested concept backpropagation on "various set of input modalities," indicating this explainability approach could potentially be applied to different AI applications—from image recognition to text analysis—making it broadly relevant for auditing AI systems across industries.