BiasFairnessExplainabilitycs.CL
Assessing the Reliability of LLMs Annotations in the Context of Demographic Bias and Model Explanation
Hadi Mohammadi, Tina Shahedi, Pablo Mosteiro, Massimo Poesio, Ayoub Bagheri, Anastasia Giachanou
July 17, 2025
Proceedings of the 6th Workshop on Gender Bias in Natural Language Processing (GeBNLP)
6 citations
Key Takeaways
-
Demographic bias in human annotations is real but limited: While demographic characteristics of human annotators do influence labeling decisions on sexism detection tasks, they account for only 8% of variance in annotations, with the actual text content being the dominant factor driving labeling choices.
-
AI personas don't reliably improve fairness: Prompting large language models with demographic personas (e.g., "annotate as a woman") often fails to improve alignment with human judgments and can sometimes perform worse than baseline models without persona guidance, suggesting this is not an effective fairness strategy.
-
AI models focus on content, not demographic proxies: Explainable AI analysis shows that language models make predictions based primarily on content-specific tokens related to sexism itself, rather than relying on demographic characteristics or their correlates, indicating the models aren't simply learning demographic shortcuts.
-
Better annotation practices matter more than persona simulation: The research suggests that developing robust annotation protocols and focusing on content-driven explanations offers a more reliable path toward fairness in NLP systems than attempting to simulate demographic perspectives through prompting.
Abstract
Understanding the sources of variability in annotations is crucial for developing fair NLP systems, especially for tasks like sexism detection where demographic bias is a concern. This study investigates the extent to which annotator demographic features influence labeling decisions compared to text content. Using a Generalized Linear Mixed Model, we quantify this inf luence, finding that while statistically present, demographic factors account for a minor fraction ( 8%) of the observed variance, with tweet content being the dominant factor. We then assess the reliability of Generative AI (GenAI) models as annotators, specifically evaluating if guiding them with demographic personas improves alignment with human judgments. Our results indicate that simplistic persona prompting often fails to enhance, and sometimes degrades, performance compared to baseline models. Furthermore, explainable AI (XAI) techniques reveal that model predictions rely heavily on content-specific tokens related to sexism, rather than correlates of demographic characteristics. We argue that focusing on content-driven explanations and robust annotation protocols offers a more reliable path towards fairness than potentially persona simulation.