FairLens: Auditing Black-box Clinical Decision Support Systems

Healthcare facilities can audit AI systems before deployment: FairLens enables healthcare organizations to test commercial clinical decision support systems on their own historical patient data to identify biases before the AI is integrated into actual clinical workflows, allowing experts to make informed decisions about whether to trust the system.

The tool identifies which patient groups are at risk: By stratifying patient data across attributes like age, ethnicity, gender, and insurance status, FairLens reveals performance disparities across different subgroups, highlighting which populations may receive systematically different (and potentially worse) clinical recommendations from the AI system.

Bias detection requires human expertise in the loop: The paper emphasizes that discovering and addressing bias in healthcare AI "should be tackled with care and involving domain experts in the loop," meaning that technical bias detection tools are most effective when clinicians and healthcare professionals actively evaluate and interpret the findings.

Explainability is essential for understanding bias: FairLens uses explainable AI (XAI) techniques to show why the model makes errors for specific patient groups—identifying which elements in clinical histories drive incorrect predictions—rather than just reporting that bias exists, enabling experts to understand the root causes.

Black-box systems can be audited without access to internal details: The methodology works on commercial "black-box" models where organizations don't have access to the underlying algorithms, making it practical for real-world healthcare settings where institutions typically cannot modify vendor systems.