Auditing Work: Exploring the New York City algorithmic bias audit regime

Vague legal definitions undermine enforcement: NYC's Local Law 144 failed to clearly define critical terms like "automated employment decision-making tools" (AEDTs) and "independent auditors," allowing companies and vendors to interpret the law in ways that weakened protections for job applicants rather than strengthening them.

Industry influence narrowed the law's scope: Through lobbying efforts, the definition of which hiring tools require auditing was narrowed, meaning many algorithmic systems used in employment decisions escaped the audit requirement entirely—demonstrating how implementation details matter as much as the law itself.

Transparency alone is insufficient: The law relied on a "transparency-driven theory of change" (the assumption that audits and public disclosure would fix bias problems), but this approach proved inadequate without stronger enforcement mechanisms and clearer accountability standards.

Practical barriers prevented effective auditing: Auditors faced significant obstacles in accessing company data and encountered disagreement about what qualifies as legitimate auditing, resulting in four different interpretations of the auditor role—creating inconsistency in how bias audits were actually conducted.

Future policies need clearer standards: Policymakers designing similar auditing regimes should establish more specific definitions, agreed-upon bias metrics, and stronger accountability measures rather than relying on industry self-regulation and voluntary compliance.