Hypothesis Testing and Machine Learning: Interpreting Variable Effects in Deep Artificial Neural Networks using Cohen's f2

Neural networks can now be tested like traditional statistics: The paper develops a framework that applies hypothesis testing to deep learning models, allowing researchers to determine whether variables have statistically significant effects—not just whether the model makes good predictions. This bridges a major gap between machine learning's predictive focus and science's need for causal inference.

Effect sizes matter as much as predictions: By adapting Cohen's f2 (a standard statistical measure) to neural networks, the framework enables researchers to quantify how large an effect is, addressing the paper's finding that "showing the magnitude of the phenomena being studied" is critical for scientific research but often missing from machine learning applications.

Black-box models can become more interpretable: The approach combines permutation-based variable importance with accumulated local effect plots and monotonicity testing to specify both what variables influence predictions and how they influence them (direction and strength), making neural network decisions more transparent to human researchers.

This could expand machine learning's use in science: The authors note that machine learning is "far less frequently used than inferential statistics" in scientific research because scientists need to test theories, not just make predictions. This framework potentially removes that barrier by making neural networks compatible with scientific hypothesis testing.

The method is model-agnostic and practical: The framework works across different neural network architectures and includes a Python implementation, suggesting it could be adopted by researchers without requiring specialized machine learning expertise.