ExplainabilitySafetyTransparencycs.AI
Explainable AI for Natural Adversarial Images
Tomas Folke, ZhaoBin Li, Ravi B. Sojitra, Scott Cheng-Hsin Yang, Patrick Shafto
June 16, 2021
arXiv.org
4 citations
2 influential
Key Takeaways
-
Human oversight of AI requires understanding how AI actually works, not just assuming it thinks like humans. The research found that people naturally assume AI decision-making mirrors their own reasoning, which can lead them to miss errors—a critical gap when humans are meant to catch AI mistakes.
-
Saliency maps (visual explanations showing which parts of an image the AI focused on) help humans spot when AI will fail. When shown saliency maps, participants were better able to predict when an image classifier would make mistakes on both normal and adversarial images, suggesting this explanation method has practical value for human oversight.
-
Showing examples of how AI classified similar images is helpful, but less effective than saliency maps. While examples did help participants predict AI errors, they were not as powerful as saliency maps alone, and combining both methods didn't produce better results than saliency maps by themselves.
-
Adversarial images reveal a real vulnerability in AI systems that human oversight could help address. The paper demonstrates that modern image classifiers can be fooled by images with subtle perturbations, but with proper explainability tools, humans may be able to catch these failures before they cause harm.
Abstract
Adversarial images highlight how vulnerable modern image classifiers are to perturbations outside of their training set. Human oversight might mitigate this weakness, but depends on humans understanding the AI well enough to predict when it is likely to make a mistake. In previous work we have found that humans tend to assume that the AI's decision process mirrors their own. Here we evaluate if methods from explainable AI can disrupt this assumption to help participants predict AI classifications for adversarial and standard images. We find that both saliency maps and examples facilitate catching AI errors, but their effects are not additive, and saliency maps are more effective than examples.