Algorithmic Bias and Data Bias: Understanding the Relation between Distributionally Robust Optimization and Data Curation

There is no one-size-fits-all solution to bias in AI: The paper demonstrates that neither Distributionally Robust Optimization (DRO)—a technical approach to minimize worst-case errors across groups—nor simply curating training data can completely solve bias problems on their own. This means organizations cannot rely on a single technical fix and must instead make deliberate choices about their specific fairness goals.

Average performance metrics hide discrimination: Machine learning systems that appear to perform well overall (low average error) often perform inconsistently across different demographic groups, potentially discriminating against underrepresented populations. This reveals why companies need to actively monitor performance across subgroups rather than relying solely on overall accuracy metrics.

Data curation and algorithmic approaches are fundamentally connected: The paper establishes theoretical equivalence between DRO and using properly weighted training data, showing that the choice between "fixing the data" versus "fixing the algorithm" is less clear-cut than often debated. Both approaches involve making value-laden decisions about which groups and outcomes matter most.

How you set up a fairness solution matters as much as which one you choose: There is no universal way to configure DRO or data curation that automatically produces "socially acceptable" results. Organizations must explicitly define what fairness means for their specific application and stakeholders, rather than assuming any technical approach will inherently be fair.

These insights extend beyond traditional bias concerns: The paper's findings about the limitations of DRO also apply to other uses of this technique, such as making AI systems robust against adversarial attacks, suggesting broader implications for how we think about building resilient AI systems.