FairnessBiasAccountabilitycs.LG
FairPlay: A Collaborative Approach to Mitigate Bias in Datasets for Improved AI Fairness
Tina Behzad, Mithilesh Kumar Singh, Anthony J. Ripa, Klaus Mueller
April 22, 2025
Proc. ACM Hum. Comput. Interact.
2 citations
Key Takeaways
-
Fairness in AI isn't one-size-fits-all: Different stakeholders have competing and sometimes incompatible ideas about what "fair" means in AI systems, so imposing a single fairness standard may not work in practice.
-
Collaboration can bridge fairness disagreements: FairPlay's user studies showed that multiple stakeholders could reach consensus on dataset debiasing in approximately five rounds of negotiation, suggesting structured dialogue is an effective path forward even without universal agreement on fairness definitions.
-
Tools matter for practical fairness work: The paper demonstrates that without systematic tools for negotiation and the ability to see real-time changes to datasets, reaching stakeholder consensus on fairness would be "highly challenging"—suggesting that technical infrastructure is essential for responsible AI governance.
-
Debiasing datasets is a collaborative process: Rather than treating bias mitigation as a purely technical problem solved by engineers alone, FairPlay positions it as a negotiation process involving multiple perspectives, which may lead to more legitimate and acceptable AI systems.
Abstract
The issue of fairness in decision-making is a critical one, especially given the variety of stakeholder demands for differing and mutually incompatible versions of fairness. Adopting a strategic interaction of perspectives provides an alternative to enforcing a singular standard of fairness. We present a web-based software application, FairPlay, that enables multiple stakeholders to debias datasets collaboratively. With FairPlay, users can negotiate and arrive at a mutually acceptable outcome without a universally agreed-upon theory of fairness. In the absence of such a tool, reaching a consensus would be highly challenging due to the lack of a systematic negotiation process and the inability to modify and observe changes. We have conducted user studies that demonstrate the success of FairPlay, as users could reach a consensus within about five rounds of gameplay, illustrating the application's potential for enhancing fairness in AI systems.