Investigating Data Interventions for Subgroup Fairness: An ICU Case Study

Combining data sources is unpredictable for fairness: When healthcare institutions merge patient data from different hospitals or departments to train AI models, simply adding more data doesn't reliably improve fairness for underrepresented groups. The study found that data addition "can both help and hurt model fairness and performance," meaning that well-intentioned efforts to expand datasets may inadvertently introduce new biases.

"More data" alone won't solve algorithmic bias: The paper challenges the common assumption that collecting additional training data automatically fixes fairness problems. Instead, the researchers found that "improvements from increased sample size are counteracted by the introduction of distribution shifts," meaning the new data often differs in important ways from the original dataset, creating new problems even as it solves others.

Intuitive data selection strategies frequently fail: Many reasonable-sounding approaches to choosing which data to add—such as simply including all available sources—proved unreliable. This suggests that organizations cannot rely on common sense alone when deciding how to combine datasets from multiple hospitals or clinical settings.

Both data and model adjustments are necessary: The most effective approach combined two strategies: carefully selecting which data to add AND adjusting the model itself after training (through "post-hoc calibration"). Neither approach alone was sufficient, indicating that fairness requires attention at multiple stages of AI development.

Healthcare AI fairness requires careful, context-specific decisions: Because real-world data sources are "less than ideal," there is no one-size-fits-all solution for achieving fairness in clinical AI systems. Organizations must thoughtfully evaluate their specific data sources rather than assuming standard practices will work.