AI auditing: The Broken Bus on the Road to AI Accountability

AI audits exist across multiple sectors but lack standardized practices: The paper identifies audits conducted by regulators, law firms, civil society organizations, journalism, academia, and consulting agencies, yet finds that the "AI audit ecosystem is muddled and imprecise," making it difficult to understand how different stakeholders approach accountability assessments.

Most AI audits fail to produce meaningful accountability outcomes: A key finding is that "only a subset of AI audit studies translate to desired accountability outcomes," suggesting that conducting an audit alone is insufficient—many audits do not lead to actual changes or accountability for AI systems' harms.

Audit design, methodology, and institutional context determine real-world impact: The paper identifies that effectiveness depends on how audits are designed and conducted, as well as the institutional power and resources of the organization performing them, indicating that not all audits are equally capable of driving accountability.

The current approach to AI accountability is fragmented and ineffective: Described as a "broken bus on the road to AI accountability," the paper suggests that without coordination and standardization across audit practices, the current patchwork of efforts is insufficient for ensuring AI systems are genuinely held accountable to the public.