Benchmarking Algorithmic Bias in Face Recognition: An Experimental Approach Using Synthetic Faces and Human Evaluation

Synthetic faces enable causal conclusions about bias, not just correlations: Unlike traditional datasets collected "in the wild," this method uses synthetic faces to isolate individual attributes (like race or gender) while keeping everything else constant. This allows researchers to definitively say an algorithm's accuracy is affected by a specific characteristic, rather than just observing that performance differs across groups in a particular dataset.

Three commercial face recognition models showed lower accuracy for Black and East Asian faces: The study's experimental evaluation of research-grade algorithms revealed measurable performance gaps, with accuracy declining for these population subgroups—a concrete finding that demonstrates bias exists in systems that may be deployed in real-world applications.

Human judgment is essential for validating algorithmic bias: The researchers used human observers to verify ground truth about whether synthetic face pairs actually looked like the same person. This human-in-the-loop approach ensures that measured algorithmic differences reflect real perceptual changes, not artifacts of the testing method.

A large public dataset can accelerate bias research: The release of 48,000 synthetic face pairs with 555,000 human annotations provides a standardized resource for other researchers to test and compare face recognition systems, potentially democratizing bias evaluation beyond well-resourced organizations.

Controlled experiments can reveal how specific visual attributes impact recognition: Beyond measuring bias, the method quantifies exactly how changes in attributes like skin tone or facial pose affect the distance scores algorithms assign to face pairs, offering insights into why bias occurs.