Do not explain without context: addressing the blind spot of model explanations

Reference data critically shapes explanations: Popular XAI techniques like Shapley values and Partial Dependence profiles produce different results depending on which dataset is used as a reference point, meaning the same model can generate contradictory explanations based on this choice alone.

Small data changes can flip conclusions: The paper demonstrates that minor shifts in the reference data distribution can cause dramatic changes in explanations, including reversing identified trends or altering final conclusions about how a model makes decisions—a particularly concerning finding for high-stakes applications.

Current explanations lack necessary context: Despite widespread adoption of explanation techniques to satisfy regulatory "right to explanation" requirements, these methods are often presented without information about the reference data used, creating a critical blind spot in model auditing and oversight.

Stakeholders may be misled: When analysts create explanations, auditors review them, and decision-makers interpret them without understanding the reference data dependency, there is significant risk that different groups will draw incorrect or conflicting conclusions about model behavior and fairness.

Robust explanations require transparency about methodology: To provide genuinely useful and trustworthy explanations, organizations must always document and communicate the reference data choices alongside any model explanation, treating this context as essential rather than optional information.