Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspaces

Neural networks make decisions based on multiple entangled factors: The paper demonstrates that current explanation methods (like heatmaps) show which input features matter, but they don't separate the different reasons why a model makes a decision. For example, a model might identify a dog in a photo based on multiple distinct visual concepts (fur texture, ear shape, body size), but traditional explanations blend these together rather than showing them separately.

New methods can disentangle explanations into distinct concepts: The researchers propose two new analysis techniques (PRCA and DRSA) that extract separate "subspaces" representing different activation patterns the neural network uses. This allows stakeholders to see not just that a model used certain features, but which distinct concepts or patterns drove the decision—making explanations more interpretable and actionable.

The approach works with existing explanation methods: Rather than replacing current popular explanation techniques like Shapley Values or Integrated Gradients, these new methods can work alongside them, suggesting they could be practically integrated into existing AI systems without major overhauls.

Focus on what models actually use, not what they ignore: Unlike traditional statistical methods that maximize variance or other general properties, these new techniques specifically maximize relevance to the model's actual prediction strategy, filtering out irrelevant activations and providing more focused, decision-relevant explanations.