Using Large Language Models to Enrich the Documentation of Datasets for Machine Learning

AI systems need better documentation to be trustworthy: The paper addresses a real gap identified by regulators like the European AI Act—datasets used to train AI models often lack clear, structured information about their origins, potential biases, and social implications, making it harder to ensure responsible AI development.

AI can help document AI: The researchers successfully used large language models (GPT-3.5 and Flan-UL2) to automatically extract key information from dataset documentation, achieving 81% accuracy with GPT-3.5, demonstrating that AI tools themselves can help create the transparency records needed for responsible AI governance.

Different AI tools have different strengths and weaknesses: While GPT-3.5 performed better overall (81% accuracy vs. 69% for Flan-UL2), it was more prone to "hallucinations" (generating false information), highlighting that no single solution is perfect and human oversight remains important when using AI for critical documentation tasks.

Machine-readable documentation enables compliance and discovery: By converting unstructured text into structured, machine-readable formats, this approach could help organizations automatically check whether their datasets comply with AI regulations and make datasets easier for researchers to find and evaluate—improving both governance and scientific practice.

Open-source tools democratize responsible AI practices: The authors released their tool and code publicly, meaning data publishers and ML practitioners can immediately start creating better documentation without needing to build these systems from scratch.