Data Annotation as Measurement

Agreement alone doesn't guarantee quality: The paper challenges the common practice of using annotator agreement as the primary measure of annotation quality, arguing that multiple annotators assigning the same label doesn't prove those labels actually measure what they're supposed to measure—a critical distinction for ensuring AI systems are built on valid data.

Annotation is measurement, not just labeling: By framing data annotation through the lens of measurement science, the paper establishes that annotation requires the same rigor as other scientific measurements: clearly defining concepts, creating standardized procedures, and validating that the process actually captures what it claims to capture.

Different annotation problems need different solutions: The research identifies five distinct sources of annotation issues (error, ambiguity, impossibility, subjectivity, and annotator identity), demonstrating that problems appearing identical in outcomes often require fundamentally different fixes—meaning one-size-fits-all quality improvement approaches are likely ineffective.

Process design matters as much as individual performance: The paper maps critical decision points across annotation workflows (task design, annotator management, quality assessment, and adjudication) that shape outcomes, suggesting that improving annotation quality requires examining systemic processes rather than focusing solely on individual annotator performance.

Current annotation practices lack scientific rigor: Based on a review of 132 studies and interviews with annotation teams, the paper reveals that annotation processes typically lack the conceptual clarity and validity checks standard in other measurement disciplines, creating a systematic vulnerability in how AI training data is created.