Error Analysis of Shapley Value-Based Model Explanations: An Informative Perspective

Shapley value explanations, a popular AI transparency tool, can be unreliable: The paper demonstrates that most current implementations of Shapley value attribution (SVA)—a method used to explain which features matter most in AI decisions—produce biased or inaccurate explanations that fail to capture true relationships between inputs and outputs.

Explanation errors fall into two categories with inherent trade-offs: The researchers identify that SVA errors stem from "observation bias" and "structural bias," and crucially, improving one type of error often worsens the other, meaning perfect explanations may be theoretically impossible with current approaches.

Common assumption-based methods tend to provide incomplete information: The paper finds that widely-used assumption-based SVA methods frequently produce "under-informative" explanations due to distribution drift—when the data used to generate explanations doesn't match the actual data the model encounters—which can mislead users about feature importance.

Different SVA methods have different failure modes: The experiments show that existing SVA implementations vary in whether they provide too much information (over-informative) or too little (under-informative), suggesting practitioners need to understand which method they're using and its specific limitations.

Better measurement tools are needed to assess explanation quality: The paper proposes new measurement approaches to quantify when and how SVA methods fail, providing a foundation for developing more reliable explainability methods in the future.