Fairness through Experimentation: Inequality in A/B testing as an approach to responsible design

A/B testing can measure fairness impacts, not just product performance: The paper proposes using the Atkinson index—an economic measure of inequality—to assess whether experiments benefit people equally across different groups. This moves beyond traditional A/B testing that only measures average outcomes, allowing companies to see if new features help some populations while leaving others behind.

"Site-wide inequality impact" reveals who gets excluded from innovation: The authors introduce a concept that measures how targeting specific subpopulations in experiments affects overall inclusiveness. This helps companies understand whether their product innovations are creating a two-tiered experience where some users benefit while others are systematically disadvantaged.

Real-world deployment at LinkedIn identified patterns in fair versus unfair innovations: After analyzing over a year of experiments across thousands of tests, the paper provides concrete learnings about which types of product changes tend to foster fairness. This suggests that fairness outcomes aren't random—certain design approaches inherently promote more inclusive results.

Responsible design requires measuring actual impact on people, not just algorithm properties: The paper emphasizes that fairness assessment must go beyond checking if an algorithm has bias in its code; companies need to experimentally measure how products actually affect different groups of people in practice, capturing real-world consequences.

Open-source tools make fairness measurement scalable and accessible: The authors provide a publicly available, scalable implementation in Spark/Scala, suggesting that measuring inequality in experiments doesn't require proprietary solutions and can be adopted by organizations of various sizes.