Do They Understand Them? An Updated Evaluation on Nonbinary Pronoun Handling in Large Language Models

Modern LLMs show improvement but gaps remain: While newer models like GPT-4o and Claude 4 demonstrate better accuracy with binary pronouns (he/she) and gender-neutral pronouns (they/them) compared to earlier versions tested in prior research, they still struggle significantly with neopronouns (such as ze/zir or ey/em), indicating that progress toward inclusive AI has been uneven.

Neopronouns represent a critical blind spot: The study found that accuracy on neopronouns and "reverse inference tasks" (where models must infer identity from pronoun usage) remains "inconsistent," suggesting that LLMs deployed in sensitive contexts may fail to properly respect or understand the full spectrum of how people identify and express gender.

Context matters: few-shot learning shows promise: The benchmark tested models under different conditions (zero-shot, few-shot, and gender identity inference), revealing that providing examples can improve performance—a finding that suggests organizations deploying these systems could improve inclusivity through better prompting strategies and training approaches.

Responsible AI deployment requires ongoing evaluation: The researchers updated the original MISGENDERED benchmark specifically because prior evaluations used "outdated models and limited evaluations," highlighting that responsible AI requires continuous reassessment as systems evolve and are deployed in "sensitive contexts where fairness and inclusivity are critical."

Model-specific differences matter for implementation: The study benchmarked five different LLMs and found model-specific variations in pronoun handling, suggesting that organizations choosing which AI system to deploy should evaluate pronoun fidelity as part of their fairness and inclusivity assessment.