A structured regression approach for evaluating model performance across intersectional subgroups

Intersectional fairness assessment is hampered by small sample sizes: When AI systems are evaluated across combinations of demographic characteristics (e.g., women of color, older men with disabilities), the number of people in each group becomes too small to reliably measure performance using standard methods, making it difficult to detect fairness problems affecting these populations.

A structured regression approach provides more reliable estimates for small groups: The paper demonstrates that using statistical regression techniques yields significantly more accurate performance measurements for tiny subgroups compared to traditional stratification methods, enabling fairer and more comprehensive fairness audits.

The method helps identify which factors drive unfair outcomes: Beyond just measuring performance gaps, the paper shows that goodness-of-fit testing can reveal which demographic characteristics or combinations thereof are the primary drivers of performance disparities, providing actionable insights for fixing AI systems.

Confidence intervals enable transparent uncertainty communication: The approach provides statistical confidence intervals around performance estimates, allowing organizations to be honest about the reliability of their fairness findings rather than presenting potentially misleading point estimates based on tiny samples.

Practical improvement demonstrated on real datasets: Testing on publicly available datasets shows the method works substantially better than current practice, suggesting it could be adopted by AI practitioners today to conduct more rigorous intersectional fairness evaluations.