RECOVER: Designing a Large Language Model-based Remote Patient Monitoring System for Postoperative Gastrointestinal Cancer Care

Stakeholder engagement is essential for responsible AI in healthcare: The researchers conducted participatory design sessions with clinical staff and patient interviews before building the system, which helped identify six major design strategies. This collaborative approach ensured the AI system reflected actual clinical needs and patient concerns rather than being designed in isolation.

LLM-based systems require careful integration with clinical guidelines: RECOVER was specifically designed to embed clinical guidelines and information needs into its conversational agent and dashboard features. This suggests that responsible AI deployment in healthcare requires deliberate architectural choices to ensure AI recommendations align with established medical protocols rather than operating independently.

Transparency and human oversight remain critical: The system included an interactive dashboard for clinical staff to monitor and review the AI's interactions with patients. This design choice reflects a responsible AI principle that AI should augment rather than replace clinical judgment, particularly in high-stakes postoperative care where complications can be life-threatening.

Patient and provider perspectives differ and both matter: By interviewing both cancer patients and clinical staff separately, the research revealed different priorities and needs. Responsible AI development in healthcare must account for these multiple perspectives rather than optimizing for a single user group.

Pilot testing with real users reveals implementation challenges: The researchers tested RECOVER with actual clinical staff and patients to assess how their design strategies worked in practice. This iterative, real-world validation approach is crucial for identifying gaps between theoretical design and actual clinical implementation before broader deployment.