"What We Can't Measure, We Can't Understand": Challenges to Demographic Data Procurement in the Pursuit of Fairness

The demographic data paradox: Practitioners working on algorithmic fairness often lack access to the demographic data they believe is necessary to detect bias, yet existing fairness toolkits and strategies almost universally require such data—creating a fundamental barrier to implementing fairness in practice.

Privacy-fairness tension has no easy solution: The research reveals a complex tradeoff between collecting demographic data needed to assess fairness and protecting individual privacy. Organizations face difficult decisions about whether it's appropriate for private companies to infer someone's demographics, and how to obtain meaningful consent for such collection.

Defining social categories is contested: There is no consensus on how to define relevant demographic categories for fairness assessments. This definitional challenge means that even when data is available, practitioners may disagree on what categories should be used to measure and address bias.

Context matters significantly: The paper found a wide spectrum of real-world scenarios—from organizations with no access to personal data at all, to others legally required to collect demographic data for discrimination assessments. This suggests that fairness solutions cannot be one-size-fits-all and must account for different regulatory and operational contexts.

Simply collecting more data isn't the answer: Rather than focusing on lowering barriers to demographic data collection, the research suggests that businesses, regulators, researchers, and community groups need to carefully consider the broader implications and develop more nuanced approaches to addressing algorithmic bias.