Large Language Models to Identify Social Determinants of Health in Electronic Health Records

LLMs can effectively extract social determinants of health from clinical notes, with fine-tuned models (Flan-T5 XL and XXL) achieving macro-F1 scores of 0.70-0.71, outperforming ChatGPT on this task—important because SDoH information is often buried in unstructured text rather than coded data.

Synthetic data generation shows promise for improving smaller models, with smaller Flan-T5 variants (base and large) gaining 12-23 percentage points in performance when trained on synthetic clinical text, suggesting a practical approach to addressing data scarcity in healthcare AI applications.

Fine-tuned models demonstrate less algorithmic bias than ChatGPT, showing greater resistance to changing predictions when race/ethnicity and gender descriptors were added to patient notes (p<0.05)—a meaningful finding for responsible deployment in clinical settings where bias could affect patient care.

Current coding systems significantly undercount social determinants, with the models identifying adverse SDoH in 93.8% of patients while ICD-10 codes captured far fewer cases, highlighting a critical gap in how healthcare systems currently document and track these important health factors.

Model performance varies significantly across different datasets, with worse performance on the MIMIC-III dataset compared to the in-hospital system, suggesting that responsible deployment requires careful validation across diverse clinical settings before implementation.