A Labeling Task Design for Supporting Algorithmic Needs: Facilitating Worker Diversity and Reducing AI Bias

Worker diversity genuinely affects labeling decisions: The study found that workers' decision-making tendencies varied based on their backgrounds, suggesting that including diverse labelers in dataset creation isn't just ethically important—it directly influences how training data is labeled and can help reduce algorithmic bias.

Community support matters as much as individual effort: The research identified that "the community that positively helps workers" was a key factor in keeping people engaged with labeling tasks. This suggests that isolated crowdworkers may produce lower-quality labels, and that building supportive communities around labeling work could improve both worker experience and AI fairness outcomes.

Machine feedback improves worker engagement and performance: Workers who received feedback from the machine during the labeling process showed better engagement. This human-in-the-loop approach—where workers see how the AI responds to their labels—appears to help workers stay motivated and potentially make more thoughtful labeling decisions.

Task design significantly impacts participation equity: The study emphasizes that how labeling tasks are designed affects whether people from different backgrounds can participate fairly. Overly complicated tasks may exclude certain workers, so thoughtful task design is necessary to achieve the diversity that helps reduce AI bias.

Labeling requires a connected ecosystem, not isolated workers: Rather than treating labelers as independent contractors, the research recommends connecting "labelers, machines, and communities" as an integrated system to support better decision-making and more equitable AI development.