Who Does What in Deep Learning? Multidimensional Game-Theoretic Attribution of Function of Neural Units

Neural networks can now be systematically analyzed to show exactly which internal units contribute to specific outputs: The paper introduces Multiperturbation Shapley-value Analysis (MSA), which can map how individual neural units influence thousands of output elements (pixels, words, or predictions) simultaneously—a capability that existing explainability methods like SHAP lack.

Large language models contain specialized "experts" organized by language: When applied to Mixtral-8x7B (a 56-billion-parameter model), MSA revealed that different neural units specialize in processing specific languages, suggesting these massive models develop internal organizational structures that weren't explicitly programmed.

Regularization techniques concentrate a model's computation into a small number of "hub" units: The research shows that when neural networks are trained with regularization (a common practice to prevent overfitting), most of the computational work gets funneled through relatively few critical units rather than being distributed across many—this has implications for both model efficiency and vulnerability.

This framework enables practical improvements to AI systems: By identifying which units matter most, MSA provides a foundation for editing neural networks (removing or modifying specific behaviors), compressing them (removing unnecessary units), and interpreting their decisions—all important for building more trustworthy and efficient AI systems.

The method works across different types of neural networks and scales: MSA was successfully applied to everything from small multi-layer perceptrons to massive language models and generative image systems, suggesting it could become a general-purpose tool for understanding how modern AI systems work.