A Guide to Feature Importance Methods for Scientific Inference

Different feature importance methods answer fundamentally different questions: The paper emphasizes that various feature importance methods (like SHAP, permutation importance, and others) have distinct interpretations, meaning selecting the wrong method for your use case can lead to incorrect conclusions about how a model works—this requires expert knowledge to get right.

Machine learning models alone cannot reveal how data is actually generated: While ML models excel at prediction, they cannot directly explain the underlying data-generating process without additional interpretation tools, which is critical when using these models for scientific understanding rather than just making predictions.

Feature importance methods only work reliably under specific conditions: The paper provides new theoretical proofs showing that feature importance methods produce valid insights into feature-target associations only when certain conditions are met—using them outside these conditions can produce misleading results about which factors actually matter.

Uncertainty quantification for feature importance is still underdeveloped: The paper identifies that methods for estimating uncertainty around feature importance results remain limited, meaning organizations often cannot confidently say how reliable their feature importance findings actually are.

Responsible AI use requires matching the right tool to the scientific question: For applications where understanding causation or the true data-generating process matters (like healthcare or policy), practitioners must carefully select appropriate feature importance methods rather than treating all methods as interchangeable.