Manipulation Risks in Explainable AI: The Implications of the Disagreement Problem

The "disagreement problem" creates vulnerability to manipulation: Because multiple valid explanations can exist for the same AI decision, bad actors could selectively present explanations that serve their interests rather than provide genuine insight into how the AI actually works, potentially misleading users about the system's true decision-making process.

Manipulation can occur at two distinct stages: Providers can either attack the underlying machine learning model or data to influence what explanations are generated, or they can manipulate the explanation phase itself by choosing which explanation to present—making the second approach potentially easier to execute without detection.

Real-world harms could include discrimination and erosion of trust: The paper identifies concrete scenarios where manipulated explanations could enable discrimination (by obscuring biased decision-making), facilitate fraud, or undermine public trust in AI systems when people discover they've been misled about how decisions affecting them were actually made.

This problem demands urgent attention before widespread deployment: The authors emphasize investigating these manipulation risks now, before explainable AI systems become standard in high-stakes domains like healthcare, criminal justice, and finance, when it will be much harder to retrofit safeguards.

Mitigation strategies are needed but require multi-stakeholder effort: While the paper proposes mitigation approaches, addressing this challenge will likely require coordination between AI developers, regulators, and users to establish standards for explanation verification and accountability mechanisms.