Understanding Machine Learning Practitioners' Data Documentation Perceptions, Needs, Challenges, and Desiderata

Current data documentation practices are fragmented and incomplete: ML practitioners at a major technology company reported that data documentation approaches are "largely ad hoc and myopic in nature," suggesting that existing efforts to document datasets lack systematic organization and forward-thinking perspective.

Practitioners need documentation tools integrated into their existing workflows: Rather than adopting standalone frameworks, participants expressed a clear need for data documentation to be "integrated into their existing tools and workflows," indicating that responsible AI practices are more likely to be adopted when they don't require significant changes to how teams already work.

Automation is essential for practical adoption: ML practitioners emphasized the need for data documentation to be "automated wherever possible," suggesting that manual documentation processes create barriers to implementation and that responsible AI frameworks must account for real-world time and resource constraints.

One-size-fits-all approaches won't work: Participants stressed that data documentation frameworks need to be "adaptable to their contexts," highlighting that different teams, projects, and organizational settings have different documentation needs and that flexible, customizable solutions are necessary for widespread adoption.

Gap between motivation and practice: The paper notes a disconnect between the responsible AI motivations behind data documentation frameworks and whether they actually meet practitioners' needs, suggesting that well-intentioned policies may fail without input from the people who must implement them.