Spotting (and Missing) Algorithmic Bias: Investigating User Understanding in a Fairness Assessment Tool

Non-technical stakeholders can identify algorithmic bias with proper tools: The study found that most participants (both HR employees and job seekers) correctly identified which groups were disadvantaged by an algorithm when using FairAware, suggesting that fairness assessment tools can be made accessible to people without technical backgrounds.

There's a dangerous gap between what people think they understand and what they actually understand: Participants reported feeling confident in their understanding of fairness metrics, but their subjective confidence didn't match their objective performance, indicating that stakeholders may make decisions about algorithmic bias while overestimating their comprehension.

Current fairness tools lack safeguards for non-expert decision-making: The authors conclude that "fairness assessment tools for non-experts are usable for identifying biases but need built-in checks on understanding before stakeholders make higher-stakes decisions"—meaning these tools should include verification mechanisms to ensure users truly grasp what they're evaluating before deploying algorithms in real-world contexts.

Domain expertise alone doesn't guarantee accurate bias assessment: HR employees and job seekers performed similarly across all measures, suggesting that professional experience in a field doesn't automatically translate to understanding algorithmic fairness, and both groups need adequate support from the tools they use.

Involving non-technical stakeholders in fairness decisions is necessary but requires careful design: Since "which biases are problematic and which metric captures them best depends on stakeholders' experience and domain knowledge," including HR professionals and affected communities in algorithmic decisions is important—but tools must be thoughtfully designed to prevent misunderstandings that could lead to flawed fairness assessments.