Holding AI to Account: Challenges for the Delivery of Trustworthy AI in Healthcare

Trustworthy AI requires more than just explanations: While the ability of AI systems to explain their decisions is important, the paper argues this alone is insufficient. True trustworthiness in healthcare depends on broader organizational accountability structures, not just individual user-AI interactions.

Organizational context matters as much as technology: The ethnographic studies of breast cancer screening and treatment planning revealed that healthcare professionals establish trust through organizational accountability—being answerable to colleagues and institutions. AI systems must be designed to fit into these existing accountability relationships rather than replacing them.

Current AI design overlooks how teams actually work: Most trustworthy AI research focuses on one person using one AI system, but the paper's findings show that healthcare decisions involve multidisciplinary teams with complex accountability chains. AI systems need to support team-based decision-making and organizational responsibility structures.

Healthcare professionals need AI to be accountable to institutions, not just users: The research demonstrates that clinicians reason about trustworthiness by considering whether an AI system can be held responsible within their organizational framework. This means AI systems must generate documentation and evidence trails that support institutional accountability, similar to how human clinicians do.

Solutions must be adaptable across different healthcare settings: The paper proposes that accountability-focused solutions developed for cancer care could potentially be redesigned and reused in other medical contexts, suggesting a pathway for scaling responsible AI practices beyond individual applications.