A qualitative field study on explainable AI for lay users subjected to AI cyberattacks

Limited engagement with explanations doesn't mean they're useless: Despite having access to interactive visualizations and explanations of how the AI worked, the 18 study participants showed only limited engagement with these XAI features. However, this doesn't indicate failure—users who did engage developed better mental models of the AI system compared to findings in previous research.

XAI can help users detect when AI systems are under attack: The study found evidence that explainable AI served as a practical tool for diagnosing cyberattacks on the smart heating system. When malicious data was injected or predictions were manipulated, users who understood the AI's normal behavior were better positioned to notice something was wrong—suggesting XAI has real security benefits beyond just user satisfaction.

Common assumptions in XAI research don't always match real-world user behavior: The researchers found that assumptions widely accepted in the XAI literature were "not aligned to reality" for actual users. This suggests that XAI design principles developed in academic settings may need adjustment when deployed in real homes with real people managing everyday technology.

Design matters: "Interpretability-by-design" showed promise: The custom Squid application was built using a simple, transparent machine learning model from the start rather than trying to explain a complex "black box" model after the fact. This approach appeared to help users develop better understanding of how the AI feature functioned.