SoK: Explainable Machine Learning for Computer Security Applications

User studies are rare in security AI research: Only 14% of studies on explainable AI for cybersecurity include user studies to evaluate whether explanations actually help analysts in real-world workflows, meaning most XAI security tools are developed without testing if they're genuinely useful to the people who would use them.

Explanations can create security vulnerabilities: The paper identifies a critical gap where explanations designed to help security analysts understand AI decisions may simultaneously expose those explanations to adversaries, who could exploit this information to evade detection systems—highlighting a tension between transparency and security.

Model designers are overlooked in security AI: The research shows that security literature focuses heavily on helping end-users and detecting attacks, but largely neglects the role of model designers who could use explainability to verify and improve AI systems before deployment, representing a missed opportunity for building more trustworthy systems.

Explainability serves multiple conflicting purposes: The paper identifies four distinct objectives for XAI in cybersecurity (user assistance, model verification, explanation robustness, and offensive use), but these goals sometimes work against each other—what helps one stakeholder may harm another, requiring careful consideration of who should access explanations.

"Interpretability by design" may be preferable to post-hoc explanations: Rather than adding explanations after an AI system is built, the paper suggests that designing inherently interpretable security systems from the start could be more effective than trying to explain complex "black box" models after the fact.