Functional requirements to mitigate the Risk of Harm to Patients from Artificial Intelligence in Healthcare

AI systems should maintain an "AI passport" documenting the system's purpose, capabilities, and limitations—similar to how medical devices are tracked—to ensure healthcare providers understand what each AI tool is designed to do and when it should not be used.

Continuous performance monitoring is essential: The paper identifies "continuous performance evaluation" and "review of retrospective/simulated cases" as key requirements to catch when AI systems degrade over time or perform poorly on new patient populations, directly addressing the risk of patient harm from AI errors.

Bias checking and explainability are critical safeguards: The proposed requirements include both "bias check" mechanisms and "eXplainable AI" to address two of the seven main risks identified by the European Parliament—bias perpetuating healthcare inequities and lack of transparency about how AI reaches its conclusions.

Multiple verification layers protect patients: The paper recommends requiring "clinicians double check" and "audit trail" capabilities, meaning AI recommendations should not be implemented without human review and all decisions should be traceable, reducing risks from both AI errors and misuse.

Data quality and security must be built-in from the start: Requirements for "data quality assessment," "encryption," and "use of field-tested libraries" address privacy, security, and accountability gaps, ensuring patient data is protected and AI systems are built on reliable foundations rather than flawed training data.