Fooling SHAP with Output Shuffling Attacks

SHAP and similar explainability tools can be manipulated to hide unfair decision-making: Researchers demonstrated that machine learning models can be modified to fool SHAP—a popular tool used to detect bias based on protected characteristics like gender or race—making discriminatory models appear fair when they are not.

These attacks don't require access to sensitive data: Unlike previous methods for fooling explainability tools, the "shuffling attacks" described in this paper work without needing access to the underlying data distribution, making them a more practical threat in real-world scenarios where data access is restricted.

The mathematical foundation of SHAP has vulnerabilities: The paper proves that Shapley values—the mathematical concept underlying SHAP—cannot theoretically detect these shuffling attacks, suggesting the problem is fundamental to how the method works, not just an implementation issue.

Current SHAP implementations offer some protection, but it's inconsistent: While the actual SHAP algorithm can detect these attacks better than simpler versions like linear SHAP, the varying degrees of effectiveness suggest that relying solely on SHAP for fairness auditing may be insufficient and risky.

Explainability tools alone are not sufficient safeguards against bias: This research highlights that using XAI methods as the primary defense against unfair AI systems has limitations, and organizations need multiple layers of oversight and testing to ensure responsible AI deployment.