ExplainabilitySecurityTrustworthinesscs.LG
eXIAA: eXplainable Injections for Adversarial Attack
Leonardo Pesce, Jiawen Wei, Gianmarco Mengaldo
November 13, 2025
arXiv.org
Leonardo Pesce, Jiawen Wei, Gianmarco Mengaldo
Explainability methods can be manipulated without changing predictions: Researchers demonstrated that images can be subtly altered to produce completely different explanations (saliency maps, integrated gradients, SHAP) while the AI model still makes the same prediction with similar confidence levels, meaning users could be misled about why a model reached its decision.
The attack requires minimal access to fool systems: Unlike previous adversarial attacks, this method only needs access to a model's predictions and explanations—not the model's internal weights or architecture—making it a practical threat to deployed AI systems across different platforms and applications.
Current explainability tools may be unreliable for high-stakes decisions: The paper raises serious concerns about using post-hoc explainability methods in safety-critical applications (healthcare, autonomous vehicles, criminal justice) where understanding the model's reasoning is essential for trust and accountability.
The vulnerability affects multiple popular explanation methods: The attack successfully manipulated explanations from three widely-used techniques (saliency maps, integrated gradients, and DeepLIFT SHAP) across different model architectures (ResNet-18 and ViT-B16), suggesting the problem is systemic rather than limited to one method.
Invisible changes can produce dramatic explanation shifts: The modifications to images are imperceptible to human eyes (validated using structural similarity metrics) yet cause substantial changes in the explanations provided to users, creating a gap between what appears trustworthy and what is actually reliable.
Post-hoc explainability methods are a subset of Machine Learning (ML) that aim to provide a reason for why a model behaves in a certain way. In this paper, we show a new black-box model-agnostic adversarial attack for post-hoc explainable Artificial Intelligence (XAI), particularly in the image domain. The goal of the attack is to modify the original explanations while being undetected by the human eye and maintain the same predicted class. In contrast to previous methods, we do not require any access to the model or its weights, but only to the model's computed predictions and explanations. Additionally, the attack is accomplished in a single step while significantly changing the provided explanations, as demonstrated by empirical evaluation. The low requirements of our method expose a critical vulnerability in current explainability methods, raising concerns about their reliability in safety-critical applications. We systematically generate attacks based on the explanations generated by post-hoc explainability methods (saliency maps, integrated gradients, and DeepLIFT SHAP) for pretrained ResNet-18 and ViT-B16 on ImageNet. The results show that our attacks could lead to dramatically different explanations without changing the predictive probabilities. We validate the effectiveness of our attack, compute the induced change based on the explanation with mean absolute difference, and verify the closeness of the original image and the corrupted one with the Structural Similarity Index Measure (SSIM).
Citation data from Semantic Scholar