TEDI: Trustworthy and Ethical Dataset Indicators to Analyze and Compare Dataset Documentation

Most datasets lack documentation on critical ethical practices: The analysis of over 100 multimodal datasets found that only a select few have documented attributes related to consent, privacy, and harmful content indicators, revealing significant gaps in transparency around how datasets handle sensitive ethical considerations.

Data collection method matters for ethical documentation: Datasets collected through crowdsourcing and direct collection approaches are more likely to document ethical practices, while web scraping—which dominates in terms of dataset scale—tends to come at the cost of ethical indicators and transparency.

TEDI provides a practical framework for comparison: The 143 fine-grained indicators introduced in this work enable systematic, empirical analysis of dataset documentation, making it possible to compare trustworthy and ethical attributes across different datasets in a standardized way.

Dataset transparency is foundational to responsible AI: The paper emphasizes that understanding multimodal dataset attributes and their ethical implications is essential for building trustworthy AI applications, yet this information remains difficult to access and compare across the field.

Scale and ethics present a trade-off, but aren't inseparable: While large-scale datasets created through scraping often lack ethical documentation, the research suggests that web scraping is "not the only viable collection method," implying that responsible alternatives exist for building substantial datasets.