AccountabilityTransparencyEthicscs.CY
Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing
Inioluwa Deborah Raji, Andrew Smart, Rebecca N. White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, Parker Barnes
January 3, 2020
FAT*
1.7K citations
109 influential
Key Takeaways
-
Internal auditing catches problems before public harm occurs: The paper identifies a critical gap where external audits only discover issues after deployment, when "emergent issues can become difficult or impossible to trace back to their source." The proposed end-to-end framework enables organizations to identify harmful repercussions during development rather than after systems are already affecting people.
-
Documentation throughout development is essential for accountability: The framework requires organizations to generate audit documents at each stage of AI system development. This creates a traceable record that connects decisions made throughout the process back to organizational values and principles, making it possible to understand how and why problems occurred.
-
Audit integrity requires systematic process design: Rather than ad-hoc reviews, the paper emphasizes that closing the "accountability gap" depends on embedding "a robust process to ensure audit integrity" into standard development practices, suggesting that accountability cannot rely on individual good intentions but must be built into organizational workflows.
-
Current practice leaves organizations unprepared: The paper notes it "remains challenging for practitioners to identify the harmful repercussions of their own systems prior to deployment," indicating that most AI developers lack adequate tools and frameworks to proactively assess potential harms—a significant gap the proposed framework aims to address.