Audit Before You Update: Recording Artifacts and the Limits of Bounded Model Updates in MIMIC-IV ICU Prediction

Data quality issues can masquerade as clinical drift: The study found that approximately half of ICU stays in 2020-2022 lacked input or procedure records, and a medication coding change made vasopressor usage appear to decline when it actually increased. This demonstrates that apparent model degradation may reflect how data are recorded rather than actual changes in clinical practice, highlighting the need to audit databases before assuming models need updating.

Bounded updates can match full retraining without sacrificing performance: When the data were properly cleaned, a parameter-bounded update strategy (retraining only certain model components) performed as well as or better than full fine-tuning in 10 of 12 test scenarios, suggesting that restrictive update protocols required by regulators like the FDA need not compromise accuracy if implemented thoughtfully.

Audit protocols are essential before implementing update strategies: The researchers developed a label-free statistical test that successfully detected data drift in every time period examined, including in basic physiological measurements, with no false alarms on clean data. This suggests that organizations should establish data auditing procedures as a prerequisite to deploying any model update strategy.

Update gains are modest even when drift is detected: Despite finding drift across all time periods, the actual performance improvements from updating models were small, suggesting that frequent model retraining may not be necessary and that careful data curation could be a more efficient approach than continuous model adaptation.