Black-box Attacks on Image Activity Prediction and its Natural Language Explanations

Explanations can be manipulated without full system access: Researchers demonstrated that attackers can fool AI explanation systems using only "black-box" access (knowing only the final output), meaning they don't need to understand how the system works internally—a more realistic threat scenario than previously studied attacks.

Multimodal explanations are vulnerable: While previous research showed that visual explanations alone could be attacked, this study reveals for the first time that AI systems generating both text and images to explain their decisions are also susceptible to adversarial manipulation, expanding the scope of known XAI vulnerabilities.

Explanations can become "unfaithful": The researchers created adversarial images that caused the model to generate explanations that don't actually reflect why it made its predictions—meaning users could receive plausible-sounding but misleading justifications for the AI's decisions.

Subtle image changes can cause major explanation shifts: The attacks used "spatially variant perturbations" (small, localized changes to images) to disrupt the link between predictions and explanations, suggesting that imperceptible image modifications could undermine trust in AI systems.

Explainability doesn't guarantee trustworthiness: This research highlights a critical gap in responsible AI: even when systems are designed to be more transparent through explanations, those explanations themselves can be compromised, requiring additional safeguards beyond explanation generation alone.