Explainability is NOT a Game

Shapley values, a popular method for explaining AI decisions, can assign importance to irrelevant features while downplaying truly relevant ones. This means explanations generated by widely-used XAI tools may mislead decision-makers about which factors actually drove a model's prediction.

The problem is particularly concerning in high-stakes domains like healthcare, criminal justice, and finance, where decision-makers rely on feature importance measures to understand and justify AI recommendations to patients, defendants, or customers.

Current explainability measures may create a false sense of understanding. Even when explanations appear scientifically rigorous (backed by Shapley value theory), they can systematically misrepresent which features matter, potentially leading to poor decisions or unjustified trust in AI systems.

The theoretical foundation of Shapley values doesn't guarantee practical reliability for feature importance. Just because a method is mathematically sound in one context doesn't mean it produces trustworthy explanations in real-world applications.

Organizations using XAI tools for accountability should reconsider their reliance on feature importance rankings alone. Decision-makers need to be aware that these explanations may not accurately reflect what the model actually learned, and should seek additional validation methods before making critical decisions.