AI2030 Responsible AI Tools Marketplace

Key Takeaways

Explainability can increase trust even when AI accuracy is lower: The study found that SOC analysts were willing to accept AI outputs with lower predictive accuracy if the explanations were perceived as relevant and evidence-backed, suggesting that transparency about why an AI made a decision matters as much as the accuracy of that decision.
One-size-fits-all explanations don't work in high-stakes security: Different analyst roles have different needs for explanations. The research emphasizes that effective XAI in SOCs requires "role-aware, context-rich" designs tailored to specific workflows rather than generic dashboard presentations of AI outputs.
Analysts want reasoning, not just results: Participants consistently expressed a strong preference for understanding the rationale behind AI decisions over simply seeing outcomes on dashboards, highlighting that meaningful explainability requires depth and context rather than simplified summaries.
Proper explainability can improve operational efficiency: When XAI designs align with actual SOC workflows, the study found they can "substantially improve practical utility," enhancing analyst comprehension, increasing triage efficiency, and supporting more confident threat responses—demonstrating real operational benefits beyond just transparency.
The "right" level of explanation matters for responsible AI deployment: The research identifies that finding the appropriate level and format of explanation is critical in high-stakes, time-pressured environments, suggesting that responsible AI implementation requires careful consideration of how much and what type of explanation users actually need.

Abstract

Explainable AI (XAI) holds significant promise for enhancing the transparency and trustworthiness of AI-driven threat detection in Security Operations Centers (SOCs). However, identifying the appropriate level and format of explanation, particularly in environments that demand rapid decision-making under high-stakes conditions, remains a complex and underexplored challenge. To address this gap, we conducted a three-month mixed-methods study combining an online survey (N1=248) with in-depth interviews (N2=24) to examine (1) how SOC analysts conceptualize AI-generated explanations and (2) which types of explanations are perceived as actionable and trustworthy across different analyst roles. Our findings reveal that participants were consistently willing to accept XAI outputs, even in cases of lower predictive accuracy, when explanations were perceived as relevant and evidence-backed. Analysts repeatedly emphasized the importance of understanding the rationale behind AI decisions, expressing a strong preference for contextual depth over a mere presentation of outcomes on dashboards. Building on these insights, this study re-evaluates current explanation methods within security contexts and demonstrates that role-aware, context-rich XAI designs aligned with SOC workflows can substantially improve practical utility. Such tailored explainability enhances analyst comprehension, increases triage efficiency, and supports more confident responses to evolving threats.

13 pages, ACM Conference on Computer and Communications Security (CCS), 2025

Key Takeaways

Explainability can increase trust even when AI accuracy is lower: The study found that SOC analysts were willing to accept AI outputs with lower predictive accuracy if the explanations were perceived as relevant and evidence-backed, suggesting that transparency about why an AI made a decision matters as much as the accuracy of that decision.
One-size-fits-all explanations don't work in high-stakes security: Different analyst roles have different needs for explanations. The research emphasizes that effective XAI in SOCs requires "role-aware, context-rich" designs tailored to specific workflows rather than generic dashboard presentations of AI outputs.
Analysts want reasoning, not just results: Participants consistently expressed a strong preference for understanding the rationale behind AI decisions over simply seeing outcomes on dashboards, highlighting that meaningful explainability requires depth and context rather than simplified summaries.
Proper explainability can improve operational efficiency: When XAI designs align with actual SOC workflows, the study found they can "substantially improve practical utility," enhancing analyst comprehension, increasing triage efficiency, and supporting more confident threat responses—demonstrating real operational benefits beyond just transparency.
The "right" level of explanation matters for responsible AI deployment: The research identifies that finding the appropriate level and format of explanation is critical in high-stakes, time-pressured environments, suggesting that responsible AI implementation requires careful consideration of how much and what type of explanation users actually need.

Abstract

13 pages, ACM Conference on Computer and Communications Security (CCS), 2025

Too Much to Trust? Measuring the Security and Cognitive Impacts of Explainability in AI-Driven SOCs

Key Takeaways

Abstract

Paper Info

Responsible AI Topics

Links

Too Much to Trust? Measuring the Security and Cognitive Impacts of Explainability in AI-Driven SOCs

Key Takeaways

Abstract

Paper Info

Responsible AI Topics

Links