Hypothesis Class Determines Explanation: Why Accurate Models Disagree on Feature Attribution

Model accuracy alone doesn't guarantee reliable explanations: The paper's core finding shows that two models with identical predictive performance can produce substantially different feature attributions, challenging the common assumption that accurate models provide trustworthy explanations for their decisions.

The type of model matters more than its performance: Across 24 datasets, models within the same class (e.g., two different tree-based models) showed strong agreement on which features matter, while different model types (trees vs. linear models) trained on identical data showed disagreement near random chance—meaning hypothesis class, not accuracy, drives explanation consistency.

This has real consequences for AI deployment and auditing: Since model selection determines which features get attributed responsibility for decisions, choosing between a tree-based and linear model could lead to different regulatory conclusions or audit results, even if both models make equally accurate predictions.

A new diagnostic tool can help identify unreliable explanations: The paper introduces the Explanation Reliability Score R(x), which can flag when explanations are unstable across different model architectures without requiring additional model training—offering a practical way to assess explanation trustworthiness.

The problem is fundamental, not easily fixed: The theoretical analysis shows this "Explanation Lottery" persists even when the data has clear interaction structures, suggesting it's a structural feature of how different model types work rather than a problem that disappears with better data or training.