RxSafeBench: Identifying Medication Safety Issues of Large Language Models in Simulated Consultation

LLMs frequently fail to identify medication safety risks in realistic clinical settings: The study found that current large language models struggle to integrate knowledge about drug contraindications and interactions, particularly when safety risks are not explicitly stated but must be inferred from patient context.

A significant gap exists between explicit and implicit medication risks: LLMs performed notably worse when medication dangers were implied rather than directly mentioned, suggesting these systems may miss subtle but critical safety issues that experienced clinicians would catch during consultations.

Comprehensive medication safety evaluation was previously impossible due to data limitations: The researchers created RxSafeBench, a benchmark containing 2,443 realistic consultation scenarios with a database of 6,725 contraindications, 28,781 drug interactions, and 14,906 indication-drug pairs—filling a critical gap in how we can test LLM safety in healthcare.

Both open-source and proprietary LLMs show medication safety vulnerabilities: The evaluation of leading models revealed widespread challenges, indicating this is not a problem limited to one type of system but rather a systemic issue across current LLM technology.

Medication safety must be a priority before deploying LLMs in clinical consultation roles: These findings demonstrate that current LLMs are not yet reliable enough for autonomous medication recommendations and highlight the need for improved safety mechanisms before such systems are used in real healthcare settings.