Challenges for Responsible AI Design and Workflow Integration in Healthcare: A Case Study of Automatic Feeding Tube Qualification in Radiology

AI alone cannot solve clinical problems: While the study found that AI can robustly detect nasogastric tube (NGT) placement from chest X-rays, the researchers discovered that "gaps remain in clinical practice integration," highlighting that technical capability is only one piece of implementing AI safely in healthcare settings.

Workflow integration requires understanding real clinical contexts: Through interviews with 15 clinical stakeholders, the team identified "trade-offs and complexities that need consideration when choosing suitable workflow stages, target users, and design configurations," demonstrating that responsible AI design must be grounded in how clinicians actually work, not just what is technically possible.

Data quality and documentation directly impact AI reliability: The research identified critical issues with "edge cases and data biases that affect model training and evaluation" and found that "data documentation practices influence data preparation and labelling," suggesting that responsible AI requires attention to often-overlooked data management practices that can introduce errors or bias.

Legal, organizational, and ethical constraints shape AI deployment: The study emphasizes the need to "balance AI benefits and risks for healthcare staff and patients within broader organizational and medical-legal constraints," indicating that responsible AI implementation must account for factors beyond pure performance metrics, including liability concerns and institutional policies.

Measuring the right outcomes matters: The researchers highlighted the importance of understanding "how to measure relevant AI outcomes reliably in future evaluations," suggesting that responsible AI requires defining success metrics that reflect actual clinical value rather than just technical accuracy.