Open Challenges on Fairness of Artificial Intelligence in Medical Imaging Applications

Bias enters medical AI systems at multiple stages: The paper identifies three critical points where fairness problems can emerge—during data collection, model training, and clinical deployment—meaning that addressing bias requires vigilance throughout the entire development pipeline, not just at one stage.

Standard fairness metrics can be misleading: The authors warn that commonly used metrics for auditing AI fairness may themselves be biased, potentially giving a false sense that a medical imaging system is fair when it actually isn't, highlighting the need for more sophisticated evaluation approaches.

"Leveling down" creates ethical dilemmas: The paper discusses a troubling phenomenon where improving fairness for disadvantaged groups by reducing overall system performance may not be an acceptable solution in medical contexts where accuracy directly impacts patient outcomes.

Performance varies unpredictably across patient populations: Different demographic groups may experience different levels of task difficulty with the same AI system, and the paper emphasizes that biases may appear in populations the developers never tested on, making it difficult to predict real-world fairness problems before deployment.

Demographic categories don't capture all sources of bias: Beyond standard attributes like race and gender, the paper notes that biases can stem from other factors not typically measured, requiring researchers to think more creatively about what hidden characteristics might cause unfair treatment in medical imaging AI.