Towards Unbiased and Accurate Deferral to Multiple Experts

Multiple experts can reduce AI bias: The paper demonstrates that systems designed to route decisions to multiple human experts—rather than relying on a single expert or the AI model alone—can significantly improve both accuracy and fairness in hybrid AI-human decision-making systems.

AI systems should recognize their own limitations: The framework allows AI models to defer cases to human experts when confidence is low, rather than making potentially biased predictions. Testing on content moderation tasks showed this approach "significantly improves the accuracy and fairness of the final predictions, compared to the baselines."

Different experts have different blind spots: The research acknowledges that each human expert brings their own domain expertise and biases. A responsible AI system should account for these individual limitations rather than treating all human input as equally reliable.

Real-world testing matters for responsible deployment: The researchers validated their approach on both synthetic datasets and real crowdsourced labels for content moderation, emphasizing that frameworks designed for human-AI collaboration need to be tested on actual human decision-making patterns, not just theoretical models.

Fairness requires intentional system design: The paper shows that fairness improvements don't happen automatically in hybrid systems—they require deliberately designing mechanisms that consider both the AI model's biases and the experts' biases when deciding when and to whom to defer decisions.