ExplainabilityTransparencyAccountabilitycs.SE
Explainability as a Compliance Requirement: What Regulated Industries Need from AI Tools for Design Artifact Generation
Syed Tauhid Ullah Shah, Mohammad Hussein, Ann Barcomb, Mohammad Moshirpour
July 12, 2025
2025 IEEE 33rd International Requirements Engineering Conference Workshops (REW)
6 citations
Key Takeaways
-
Manual validation negates efficiency gains: The study found that when AI tools lack explainability, practitioners in safety-critical industries must perform extensive manual validation of AI-generated outputs, which often eliminates the anticipated efficiency benefits that motivated AI adoption in the first place.
-
Regulatory compliance is a critical barrier: Non-explainable AI outputs create "regulatory compliance risks" in regulated industries, meaning companies cannot easily demonstrate to regulators how and why the AI made specific decisions—a requirement for safety-critical sectors where transparency and traceability are legally mandated.
-
Lack of trust undermines adoption: The research reveals that unexplainable AI outputs "reduce stakeholder trust," suggesting that even when tools work technically, practitioners and teams are reluctant to rely on them without understanding the reasoning behind the AI's decisions.
-
Domain expertise is lost in translation: Current AI tools "struggle to handle domain-specific terminology," meaning they cannot adequately understand industry-specific language and concepts, forcing practitioners to either rework outputs or abandon the tools entirely rather than risk safety or compliance issues.
-
Specific solutions exist: The study identifies concrete improvements needed, including "source tracing" (showing where the AI pulled information from), "clear justifications for tool-generated decisions," domain-specific customization, and built-in compliance validation features—providing a practical roadmap for making AI tools viable in regulated industries.
Abstract
Artificial Intelligence (AI) tools for automating design artifact generation are increasingly used in Requirements Engineering (RE) to transform textual requirements into structured diagrams and models. While these AI tools, particularly those based on Natural Language Processing (NLP), promise to improve efficiency, their adoption remains limited in regulated industries where transparency and traceability are essential. In this paper, we investigate the explainability gap in AI-driven design artifact generation through semi-structured interviews with ten practitioners from safety-critical industries. We examine how current AI-based tools are integrated into workflows and the challenges arising from their lack of explainability. We also explore mitigation strategies, their impact on project outcomes, and features needed to improve usability. Our findings reveal that non-explainable AI outputs necessitate extensive manual validation, reduce stakeholder trust, struggle to handle domain-specific terminology, disrupt team collaboration, and introduce regulatory compliance risks, often negating the anticipated efficiency benefits. To address these issues, we identify key improvements, including source tracing, providing clear justifications for tool-generated decisions, supporting domain-specific adaptation, and enabling compliance validation. This study outlines a practical roadmap for improving the transparency, reliability, and applicability of AI tools in requirements engineering workflows, particularly in regulated and safety-critical environments where explainability is crucial for adoption and certification.
Paper Info
arXiv ID2507.09220
PublishedJuly 12, 2025
Citations
6
Relevance92%
Venue2025 IEEE 33rd International Requirements Engineering Conference Workshops (REW)
TypeJournalArticle, Conference
Categorycs.SE
Responsible AI Topics
ExplainabilityTransparencyAccountability
Links
Citation data from Semantic Scholar