Detecting Backdoor Poisoning Attacks on Deep Neural Networks by Heatmap Clustering

Backdoor attacks on AI systems are a real threat: Neural networks can be manipulated through "poisoning attacks" where malicious data is inserted into training datasets to cause the AI to make fraudulent predictions, highlighting a significant security vulnerability in AI systems.

Detection methods can identify compromised AI models: The paper demonstrates that two clustering-based detection approaches—Heatmap Clustering and Activation Clustering—can successfully identify poisoned data within datasets, offering practical tools to catch these attacks before deployment.

Explainable AI methods strengthen security: Heatmap Clustering, which uses explainable AI techniques (Layer-wise Relevance Propagation) to visualize what the neural network "sees," outperformed the alternative method, suggesting that making AI systems more interpretable can also make them more secure.

Different attack types require different defenses: The paper found that detection effectiveness varies depending on the attack method used, with label-consistent poisoning attacks being particularly challenging—indicating that robust AI security requires defenses tailored to multiple threat scenarios.

No single perfect solution exists yet: While both detection methods showed promise, neither achieved perfect performance across all attack types, suggesting that organizations need layered security approaches rather than relying on any single detection method.