What Shapes Participant Data Quality? A Scoping Review and Case Study of Crowdsourced Webcam Eye Tracking in AI Interviews

Data quality in crowdsourced eye tracking is unpredictable and poorly documented: The scoping review of studies from 2011-2025 found "fragmented reporting and a lack of established quality benchmarks," meaning researchers currently lack clear standards for assessing whether data collected through webcams is reliable enough to trust.

Technical and behavioral factors significantly affect data reliability: The case study of 205 AI fairness interviews identified specific predictors of quality—including higher fixation counts, shorter session lengths, and operating system choice—suggesting that data quality isn't random but can be anticipated and potentially improved through study design.

Hardware diversity and uncontrolled environments create systematic quality problems: The paper identifies that webcam-based eye tracking's main weakness is the lack of controlled conditions across participants' different devices and home environments, which undermines the consistency needed for fair AI research.

Current practices lack transparency and replicability: The fragmented reporting standards mean that findings from crowdsourced eye-tracking studies may not be reproducible or comparable across research, raising concerns about whether conclusions about AI fairness or other topics are actually reliable.

Actionable improvements are possible with better guidelines: The authors provide specific recommendations to enhance reliability and transparency, suggesting that establishing quality benchmarks and reporting standards could make crowdsourced eye tracking a more trustworthy method for reaching diverse populations in responsible AI research.