Evaluating the Promise and Pitfalls of LLMs in Hiring Decisions

General-purpose AI models underperform in hiring decisions: State-of-the-art LLMs from major companies (OpenAI, Google, Anthropic, Meta, Deepseek) achieved a ROC AUC score of 0.77 for predicting good job candidates, compared to 0.85 for a specialized hiring model, meaning they are significantly less accurate at identifying qualified candidates.

Off-the-shelf LLMs can amplify existing societal biases: The study found that general LLMs showed concerning racial bias in hiring recommendations, with impact ratios as low as 0.809 (meaning some racial groups were selected at less than 81% the rate of others), compared to 0.957 for the specialized model—a substantial fairness gap that could perpetuate discrimination.

Specialized, audited models are essential for high-stakes decisions: Domain-specific models trained with bias mitigation in mind performed dramatically better on both accuracy and fairness metrics than applying general-purpose AI tools to hiring, suggesting that critical decisions about people's livelihoods require tailored solutions with rigorous testing.

Pretraining biases in LLMs pose hidden risks: The paper identifies that biases absorbed during LLM training on internet data can be unknowingly transferred into hiring decisions unless specific safeguards are implemented, highlighting a systemic risk when companies adopt AI tools without understanding their limitations.

Bias auditing must be mandatory before deployment: The research emphasizes that organizations cannot simply deploy commercial AI systems in hiring without first conducting fairness testing across demographic groups—a critical step that appears to be frequently skipped in practice.