AI2030 Responsible AI Tools Marketplace

Key Takeaways

Explainable AI systems can be manipulated just like the models they explain: The paper demonstrates that popular XAI methods like SHAP, LIME, and Integrated Gradients are vulnerable to adversarial attacks that alter their explanations, meaning users cannot always trust the reasoning provided by these transparency tools.
Three main attack types threaten XAI reliability: The research identifies fairwashing (making unfair decisions appear fair), manipulation of explanations (changing what the system claims influenced its decision), and backdoor attacks (hidden triggers that alter explanations) as specific vulnerabilities in cybersecurity applications.
Real-world security applications are at risk: The attacks were tested on practical cybersecurity scenarios including phishing detection, malware identification, intrusion detection, and fraudulent website detection, suggesting that critical security systems relying on XAI for decision-making could be compromised.
Trust in AI transparency may be misplaced without additional safeguards: While XAI methods were designed to increase trust and accountability in AI systems, this paper reveals that trust placed in their explanations could be exploited by attackers, creating a false sense of security and understanding.
Urgent need for more resilient explanation methods: The paper calls for immediate attention to developing XAI methods that are resistant to adversarial attacks, suggesting current transparency tools require significant improvements before they can reliably support responsible AI deployment in high-stakes domains.

Abstract

Explainable Artificial Intelligence (XAI) has aided machine learning (ML) researchers with the power of scrutinizing the decisions of the black-box models. XAI methods enable looking deep inside the models' behavior, eventually generating explanations along with a perceived trust and transparency. However, depending on any specific XAI method, the level of trust can vary. It is evident that XAI methods can themselves be a victim of post-adversarial attacks that manipulate the expected outcome from the explanation module. Among such attack tactics, fairwashing explanation (FE), manipulation explanation (ME), and backdoor-enabled manipulation attacks (BD) are the notable ones. In this paper, we try to understand these adversarial attack techniques, tactics, and procedures (TTPs) on explanation alteration and thus the effect on the model's decisions. We have explored a total of six different individual attack procedures on post-hoc explanation methods such as SHAP (SHapley Additive exPlanations), LIME (Local Interpretable Model-agnostic Explanation), and IG (Integrated Gradients), and investigated those adversarial attacks in cybersecurity applications scenarios such as phishing, malware, intrusion, and fraudulent website detection. Our experimental study reveals the actual effectiveness of these attacks, thus providing an urgency for immediate attention to enhance the resiliency of XAI methods and their applications.

10 pages, 9 figures, 4 tables

Key Takeaways

Explainable AI systems can be manipulated just like the models they explain: The paper demonstrates that popular XAI methods like SHAP, LIME, and Integrated Gradients are vulnerable to adversarial attacks that alter their explanations, meaning users cannot always trust the reasoning provided by these transparency tools.
Three main attack types threaten XAI reliability: The research identifies fairwashing (making unfair decisions appear fair), manipulation of explanations (changing what the system claims influenced its decision), and backdoor attacks (hidden triggers that alter explanations) as specific vulnerabilities in cybersecurity applications.
Real-world security applications are at risk: The attacks were tested on practical cybersecurity scenarios including phishing detection, malware identification, intrusion detection, and fraudulent website detection, suggesting that critical security systems relying on XAI for decision-making could be compromised.
Trust in AI transparency may be misplaced without additional safeguards: While XAI methods were designed to increase trust and accountability in AI systems, this paper reveals that trust placed in their explanations could be exploited by attackers, creating a false sense of security and understanding.
Urgent need for more resilient explanation methods: The paper calls for immediate attention to developing XAI methods that are resistant to adversarial attacks, suggesting current transparency tools require significant improvements before they can reliably support responsible AI deployment in high-stakes domains.

Abstract

10 pages, 9 figures, 4 tables

Explainable but Vulnerable: Adversarial Attacks on XAI Explanation in Cybersecurity Applications

Key Takeaways

Abstract

Paper Info

Responsible AI Topics

Links

Explainable but Vulnerable: Adversarial Attacks on XAI Explanation in Cybersecurity Applications

Key Takeaways

Abstract

Paper Info

Responsible AI Topics

Links