Exploring the Requirements of Clinicians for Explainable AI Decision Support Systems in Intensive Care

Clinicians need explanations tailored to their complex decision-making process: The study found that ICU decision-making relies on "a wide range of factors" beyond what AI systems typically consider. This means responsible AI in healthcare must be designed to complement clinicians' holistic reasoning rather than replace it, requiring explanations that account for the full context clinicians use to make decisions.

One-size-fits-all AI explanations won't work in intensive care: The research involved clinicians with "various roles and experience levels," suggesting that different team members (doctors, nurses, specialists) need different types of explanations from the same AI system. Responsible AI deployment requires customizable explanations rather than generic outputs.

Patient complexity creates barriers to AI-human collaboration: The study identified that "the complexity of patient state is challenging for shared decision-making," indicating that AI systems must be designed to handle—and clearly communicate about—the inherent uncertainty and multifaceted nature of critical care. This is essential for building appropriate trust rather than over-reliance on AI recommendations.

Clinical input is essential before deployment: Rather than building AI systems and then asking clinicians to adapt, this research demonstrates that gathering requirements from actual ICU staff through interviews should inform system design from the start. This participatory approach helps ensure AI tools are genuinely useful and safe in practice.

Trustworthiness depends on transparency about AI limitations: The paper emphasizes concerns about "trustworthiness" arising from "increasingly complex and less human-interpretable" AI models, suggesting that responsible AI in critical care requires honest communication about what the system can and cannot do reliably.