Human-Centered Development of an Explainable AI Framework for Real-Time Surgical Risk Surveillance

User needs must drive AI design in healthcare: The study found that successful AI adoption in surgery depends on understanding how doctors actually make decisions and work, not just building accurate algorithms. Researchers conducted 11 co-design sessions with 20 surgeons and anesthesiologists to understand their workflows before and during prototype development.

Explainability is essential for clinical trust: One of five major themes identified was "trustworthy considerations," indicating that physicians need to understand why an AI system makes its recommendations—not just receive a risk prediction. This transparency appears critical for doctors to incorporate AI into their decision-making rather than reject it.

Real-time, data-integrated predictions address a genuine clinical gap: Physicians valued that MySurgeryRisk could process large volumes of patient data and deliver risk assessments in real-time for nine different post-operative complications, suggesting the tool fills a practical need in their current workflow that manual assessment cannot meet.

Iterative feedback from end-users improves AI tools: By involving actual surgical teams in the design process and collecting their feedback on the prototype interface and features, the researchers ensured the final tool would be usable and relevant—a model that contrasts with AI systems deployed without such consultation.

Diverse perspectives strengthen responsible AI development: The study included surgeons and anesthesiologists at varying career stages, suggesting that incorporating voices from different experience levels and specialties helps identify potential issues and use cases that a single perspective might miss.