Beyond the Single-Best Model: Rashomon Partial Dependence Profile for Trustworthy Explanations in AutoML

Single-model explanations can be misleading: The research found that explanations from a single "best" model cover less than 70% of the partial dependence profiles from other near-optimal models in most cases, meaning users relying on one model's explanation are missing important information about how features actually affect predictions.

Multiple good models often disagree on feature effects: The Rashomon set approach reveals that different high-performing models can interpret feature relationships differently. By aggregating explanations from multiple near-optimal models rather than just one, the framework shows where models agree and disagree, providing a more complete picture of explanation uncertainty.

Uncertainty in AI explanations matters for high-stakes decisions: The paper emphasizes that in critical domains like healthcare or finance, users need to understand not just what a model predicts, but how confident we should be in the explanation of why it makes that prediction. Showing disagreement between models helps communicate this uncertainty.

Current AutoML systems prioritize accuracy over explanation reliability: Automated machine learning typically selects a single best-performing model without considering whether its explanations are robust or representative. This framework demonstrates that trustworthy AI requires evaluating explanation consistency across multiple viable models, not just model performance.

A practical solution for more transparent AI: The Rashomon PDP framework provides a concrete method to make AutoML systems more transparent by visualizing confidence intervals and areas of disagreement in feature effects, making it easier for non-technical stakeholders to understand the limitations of model interpretations.