Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collection

One-time data collection is insufficient: The paper identifies that datasets are typically collected once and then reused for different purposes, which can lead to unfair, biased, or inaccurate AI outcomes. This practice fails to account for how data quality changes over time or across different applications.

Dataset quality requires ongoing measurement: Rather than treating data collection as a finished task, the authors propose a systematic approach using specific metrics to continuously scrutinize and measure data quality. This "collect, measure, repeat" methodology helps identify problems like ambiguous annotations, errors, and lack of generalization across domains.

Documentation alone is insufficient for responsible AI: While the paper acknowledges that dataset documentation (origin, development process, ethical considerations) is important, it emphasizes that documentation must be paired with active measurement and monitoring of data reliability to truly ensure fairness and robustness in AI systems.

Data reliability has internal and external dimensions: The proposed framework measures both "internal reliability" (quality within a dataset) and "external stability" (how well data generalizes over time and across different contexts), recognizing that responsible AI requires attention to both aspects.

Systematic metrics enable better AI governance: By providing a granular set of measurements validated across nine existing datasets, the paper offers practical tools for organizations to assess whether their data collection practices meet responsible AI standards before deploying machine learning models in high-stakes domains.