Data Annotation Quality Problems in AI-Enabled Perception System Development

Annotation errors are systematic and widespread: The study identified 18 recurring types of annotation errors across three dimensions (completeness, accuracy, and consistency) in autonomous vehicle development, indicating these are not isolated mistakes but predictable failure modes that occur across multiple organizations.

The supply chain amplifies quality problems: Because annotation work spans multiple organizations in the automotive supply chain, errors can emerge and spread across company boundaries. The research reveals that annotation quality is not just a technical problem but a multi-organizational coordination challenge requiring shared standards and communication protocols.

Three critical error categories threaten safety: Completeness errors (like missing edge cases), accuracy errors (like mislabeling), and consistency errors (like disagreement between annotators) each pose distinct risks to AI system reliability—suggesting that responsible AI development requires monitoring across all three dimensions, not just one.

Industry practitioners validated a practical diagnostic tool: The taxonomy was confirmed as useful by experts for root-cause analysis, supplier reviews, and improving guidelines, functioning like a "failure-mode catalogue" similar to established engineering safety practices (FMEA), making it actionable for companies developing autonomous systems.

Better annotation governance is essential for safety: The findings suggest that responsible AI-enabled perception systems require not just better technology, but improved processes: clearer instructions, feedback loops, consistent hand-offs between teams, and systematic quality reviews across the supply chain.