Exploring Learned Representations of Neural Networks with Principal Component Analysis

Neural networks use far fewer features than they learn: The research found that deep neural networks trained on image recognition only need about 20% of the information they actually learn in certain layers to maintain high accuracy, suggesting significant redundancy in how these systems represent information internally.

A small number of key patterns drive network decisions: Across all layers studied, approximately 100 principal components (out of thousands of learned features) completely determined classifier performance, indicating that neural networks may rely on a surprisingly small set of core patterns to make decisions.

Simpler models can replicate neural network behavior: The researchers successfully created interpretable substitute models (particularly an affine linear model) that mimicked how the neural network made decisions based on learned representations, offering a potential tool for understanding what features influence AI predictions.

Understanding where networks "collapse" their learning: The work provides evidence for "neural collapse"—a phenomenon where neural networks compress their learned representations into simpler structures—and offers a method to identify where this occurs, which could help researchers understand how networks organize information.

Explainability through layer-by-layer analysis: By examining how information flows through different layers of the network using PCA, this approach demonstrates a practical method for making "black box" neural networks more transparent and interpretable to non-experts.