Threading the Needle of On and Off-Manifold Value Functions for Shapley Explanations

Both current approaches to explaining AI models have significant flaws: On-manifold value functions (which stay close to real data) violate important mathematical properties and are computationally expensive, while off-manifold value functions (which test broader scenarios) evaluate models on unrealistic data the system was never trained on—meaning neither approach is fully trustworthy for explaining AI decisions.

Adversarial attacks can manipulate feature importance explanations: Both types of value functions are vulnerable to manipulation in low-density regions, suggesting that bad actors could potentially game Shapley-based explanations to hide or misrepresent which features actually drive a model's decisions.

A new "Joint Baseline" approach balances competing demands: The researchers propose a mathematically unique solution that respects both the actual data distribution and the model's behavior while being robust to adversarial perturbations, offering a more principled way to explain AI systems.

The choice of explanation method matters significantly for responsible AI: Since different value functions can produce different feature importance rankings, organizations using Shapley explanations need to carefully consider which method they use—the choice isn't merely technical but affects whether stakeholders get accurate, manipulable, or misleading explanations of model behavior.