Perspectives in Play: A Multi-Perspective Approach for More Inclusive NLP Systems

Aggregating all annotator opinions into a single "ground truth" can erase minority perspectives, particularly in subjective tasks like hate speech and stance detection. This paper demonstrates that traditional methods of averaging human judgments systematically underrepresent certain viewpoints, raising concerns about whose values get encoded into AI systems.

AI systems trained on "soft labels" that preserve human disagreement perform better than those trained on averaged labels, achieving higher F1 scores while more accurately reflecting the actual distribution of human opinions. This suggests that acknowledging disagreement isn't just more inclusive—it's also more technically sound.

Subjective AI tasks inherently involve value judgments tied to annotators' backgrounds and experiences, not objective facts. The paper emphasizes that labels in areas like irony detection and stance detection reflect diverse life experiences and preferences, meaning developers must consciously decide whose perspectives to include rather than pretending objectivity exists.

Lower model confidence in subjective tasks may be appropriate rather than a flaw. The authors note that tasks with high human disagreement produce models with lower confidence, which could be a feature rather than a bug—signaling to users when AI systems are navigating genuinely contested terrain rather than falsely claiming certainty.

Responsible AI development requires moving beyond single-label systems toward "perspective-aware" models that acknowledge multiple valid viewpoints exist, particularly for socially sensitive applications like content moderation and bias detection.