Auditing the Use of Language Models to Guide Hiring Decisions

LLMs show measurable bias in hiring decisions: The study found "moderate race and gender disparities" when large language models were used to evaluate teaching job applications, meaning the models rated candidates differently based on demographic indicators like names—a finding that persists even when researchers changed which application materials were provided to the models.

Current regulations lack clear auditing guidance: The paper identifies a critical gap in existing regulatory frameworks, noting that "current regulations—as well as the scientific literature—provide little guidance on how to conduct these assessments," highlighting that policymakers are moving faster than the field's ability to establish standardized testing methods.

Correspondence experiments offer a practical auditing approach: The researchers adapted a well-established method from social science research (testing for bias in human decision-making) to evaluate AI systems, demonstrating that existing tools can be repurposed to detect algorithmic bias in hiring contexts, though the paper acknowledges important limitations to this approach.

Bias patterns are consistent across different inputs and framings: The disparities the team observed remained "largely robust to varying the types of application material input to the models, as well as the framing of the task," suggesting the bias is a deeper characteristic of these systems rather than an artifact of how they're prompted.

Auditing methods have important limitations: The authors conclude by cautioning that correspondence experiments, while useful, have constraints for auditing algorithms—implying that no single testing method provides complete assurance of fairness in AI hiring tools.