A Framework for Benchmarking Fairness-Utility Trade-offs in Text-to-Image Models via Pareto Frontiers

Current fairness evaluations are unreliable: The paper identifies that existing methods for assessing fairness in image-generating AI rely on subjective human judgment and narrow comparisons, making it difficult to consistently and reproducibly evaluate whether these systems are actually reducing bias.

There's a measurable trade-off between fairness and image quality: The research demonstrates that making text-to-image models fairer (less biased) often comes at a cost to visual quality, and vice versa. Using their framework, they found that most models' default settings are suboptimal—meaning better configurations exist that could improve fairness without sacrificing as much quality.

A new benchmarking method enables fair comparison: The paper proposes using "Pareto-optimal frontiers" as a standardized way to evaluate multiple text-to-image models (including Stable Diffusion, SDXL, and FLUX) on fairness-utility trade-offs, allowing organizations to objectively compare which models and settings work best for their needs.

Better configurations are readily available: The analysis shows that simply adjusting hyperparameters (technical settings) in existing models can improve fairness outcomes, suggesting that responsible AI improvements don't always require building entirely new systems—they may just require better tuning of current ones.

Reproducibility matters for responsible AI: By replacing subjective visual inspections with quantitative metrics and a systematic framework, the paper addresses a key barrier to responsible AI: the ability to verify fairness improvements and hold AI systems accountable through consistent, replicable evaluation methods.