Cross-modality debiasing: using language to mitigate sub-population shifts in imaging

Multi-modal AI models like CLIP have built-in fairness advantages: The research found that vision-language models (which process both images and text) naturally show better performance across different population groups compared to image-only models, suggesting that combining different types of data can improve fairness.

Fine-tuning can undermine fairness gains: When these multi-modal models are customized for specific tasks through fine-tuning, they lose their inherent fairness protections, demonstrating that the process of adapting AI systems to new uses can inadvertently reintroduce bias.

Language can be used as a debiasing tool: By using natural language descriptions to guide how the model processes images, researchers were able to reduce bias and improve performance for underrepresented groups, offering a practical technique for making AI systems more equitable.

This approach reduces performance instability across populations: The method not only improved worst-case performance for disadvantaged groups but also made the model's performance more consistent across different sub-populations, addressing a key measure of algorithmic fairness.

Cross-modality approaches offer a new debiasing strategy: Rather than only adjusting image processing, leveraging connections between different data types (vision and language) provides a novel way to mitigate bias that could be applied to other multi-modal AI systems.