Is invariance all you need for algorithmic fairness? Removing demographic information can create new bias

Removing all demographic information from AI models can backfire: The paper demonstrates both mathematically and empirically that enforcing "demographic invariance" (removing all traces of demographic data from a model's internal workings) can actually increase bias rather than reduce it, contrary to common industry practice.

Context matters: correlation between demographics and outcomes is key: When demographic characteristics are genuinely correlated with what the model is trying to predict, some encoding of that information is necessary. Completely stripping it out forces the model to find indirect workarounds that may be less transparent and potentially more harmful.

Two types of invariance have different fairness implications: The researchers distinguish between "marginal" invariance (removing demographics entirely) and "class-conditional" invariance (removing demographics only within groups with the same outcome). Class-conditional invariance aligns with the fairness concept of "equalized odds," while marginal invariance aligns with "demographic parity"—and they have different real-world effects.

One-size-fits-all fairness approaches are insufficient: The findings suggest that simply enforcing demographic invariance is "neither desirable nor sufficient for fairness," indicating that fairness requires more nuanced, context-specific approaches rather than blanket technical fixes.

Testing across diverse datasets strengthens the finding: The conclusions are supported by evaluation across five tabular datasets and two chest X-ray imaging datasets, suggesting the problem isn't limited to one type of data or application domain.