Relating the Partial Dependence Plot and Permutation Feature Importance to the Data Generating Process

Popular AI interpretation tools have hidden limitations: Partial dependence plots and permutation feature importance—widely used methods to explain what machine learning models learn—can systematically deviate from the true relationships in the data due to statistical biases and model variability, meaning practitioners may draw incorrect conclusions about how their models actually work.

Current methods lack theoretical grounding: The paper formalizes these interpretation techniques as statistical estimators for the first time, revealing that without proper theory connecting them to the underlying data generating process, there's no principled way to know how reliable these explanations are.

Model uncertainty matters for interpretability: The researchers show that different versions of the same model (trained on different data samples) can produce different interpretations, highlighting that model variance is a significant but often-overlooked source of error in AI explanations.

Better confidence measures are needed: The paper proposes improved methods (learner-PD and learner-PFI) that provide corrected variance estimates and confidence intervals, enabling practitioners to quantify uncertainty in their model interpretations rather than treating them as definitive facts.

Responsible AI requires acknowledging interpretation uncertainty: For organizations using machine learning to make consequential decisions, this work underscores the importance of not over-trusting interpretation methods and instead treating model explanations as estimates with measurable uncertainty, similar to how statisticians approach inference.