TRIP: A Nonparametric Test to Diagnose Biased Feature Importance Scores

Permutation feature importance, a widely-used method for explaining AI model decisions, can give misleading results when features in the data are related to each other. The paper shows that this popular technique creates "unrealistic observations" when shuffling correlated features, leading to incorrect conclusions about which features actually matter for predictions.

The paper introduces TRIP, a new diagnostic test that can detect when feature importance scores are unreliable. This test requires "minimal assumptions" and works across different types of models, making it a practical tool for practitioners to verify whether their model explanations can be trusted before relying on them for decision-making.

The problem occurs because permutation feature importance can cause models to make predictions outside their normal operating range. When features are dependent on each other and one is artificially shuffled, the model encounters data combinations it was never trained on, leading to unreliable importance estimates—a phenomenon the authors call "model extrapolation."

This work addresses a critical gap in responsible AI: we often cannot trust explanations of complex models without verification. Since feature importance scores are frequently used to justify high-stakes decisions (in hiring, lending, healthcare, etc.), having a way to diagnose when these explanations are misleading is essential for ensuring AI systems are genuinely interpretable and trustworthy.