"Faithful to What?" On the Limits of Fidelity-Based Explanations

High fidelity doesn't guarantee faithful explanations: The paper demonstrates that surrogate models can achieve high fidelity scores (meaning they accurately mimic a neural network's predictions) while simultaneously failing to capture the actual patterns in the underlying data that make the network useful. This means a popular metric for evaluating AI explanations may be misleading.

Fidelity measures the wrong thing: Current fidelity-based evaluation asks "does this explanation match what the model does?" rather than "does this explanation reveal what the model learned about the real-world task?" The paper shows these are fundamentally different questions, with important implications for how we validate explanations.

Surrogates can underperform simple alternatives: In several real-world cases tested, high-fidelity surrogate explanations actually performed worse than basic linear models trained directly on the data. This reveals a critical gap: explaining a complex model's behavior doesn't necessarily reveal task-relevant insights that could be obtained more simply.

The linearity score (λ) offers a diagnostic tool: The paper introduces a new metric that measures how much of a neural network's advantage comes from learning nonlinear patterns versus simply fitting noise. This helps distinguish between models that genuinely capture complex task structure and those that merely overfit.

Practitioners should reconsider explanation evaluation: For responsible AI deployment, organizations relying on fidelity scores to validate that explanations are trustworthy may have false confidence. The paper suggests the need for evaluation methods that verify explanations capture task-relevant structure, not just model behavior.