Evaluating LLM-Contaminated Crowdsourcing Data Without Ground Truth

Human feedback is essential for trustworthy AI, but crowdsourcing quality is at risk: The paper highlights that as AI systems increasingly rely on human feedback to improve their behavior, workers are now using LLMs to generate responses instead of providing genuine human input, potentially undermining the entire foundation of responsible AI development.

Traditional AI detection methods don't work for simple labeling tasks: Existing approaches to catch LLM-generated content focus on analyzing full text, making them ineffective for common crowdsourcing work like multiple-choice annotations—the paper addresses this gap by developing a method that works without needing ground truth labels.

A new detection approach uses worker agreement patterns: Rather than analyzing the content itself, the proposed method identifies suspicious LLM use by examining whether workers' answers correlate in unusual ways, even accounting for cases where multiple workers might collude using the same LLM.

The method works without requiring correct answers upfront: Unlike previous approaches, this technique doesn't need a dataset of verified correct answers to function, making it practical for real-world crowdsourcing where ground truth may be expensive or difficult to obtain.

The approach successfully identifies "low-effort cheating" in real datasets: The paper demonstrates empirical effectiveness on actual crowdsourcing data, suggesting this method could help organizations protect the quality of human feedback used to train and align AI systems responsibly.