Mohammmad Tahaei, Daricia Wilkinson, Alisa Frik, Michael Muller, Ruba Abu-Salma, Lauren Wilcox
Surveys on AI often reflect Western biases rather than global perspectives: The paper's analysis of 44 studies found that public surveys on AI topics are "vulnerable to specific Western knowledge, values, and assumptions in their design," meaning findings presented as universal public opinion may actually represent primarily Western viewpoints and priorities.
Ethical concepts in AI surveys lack cultural grounding: The research identifies that surveys frequently position ethical concepts and societal values without sufficient consideration of how these concepts are understood differently across cultures, potentially leading to misaligned conclusions about what the public actually values in AI systems.
Survey deployment strategies receive insufficient critical examination: The paper found that existing AI surveys demonstrate a "lack of sufficient critical discourse surrounding deployment strategies," suggesting researchers often fail to adequately question how and why surveys are being used as tools for public engagement in AI governance.
Transparency and reporting standards are inconsistent across AI surveys: The systematic review revealed "inconsistent forms of transparency in their reporting," meaning readers cannot reliably assess the quality, limitations, or potential biases of different surveys—making it difficult to compare findings or trust conclusions.
Surveys alone may be insufficient for meaningful public engagement: The authors argue that the field should "recognize the limitations of surveys for meeting the goals of engagement," suggesting that surveys should be complemented with other methods to genuinely capture diverse public values in AI development and governance.
Calls for engagement with the public in Artificial Intelligence (AI) research, development, and governance are increasing, leading to the use of surveys to capture people's values, perceptions, and experiences related to AI. In this paper, we critically examine the state of human participant surveys associated with these topics. Through both a reflexive analysis of a survey pilot spanning six countries and a systematic literature review of 44 papers featuring public surveys related to AI, we explore prominent perspectives and methodological nuances associated with surveys to date. We find that public surveys on AI topics are vulnerable to specific Western knowledge, values, and assumptions in their design, including in their positioning of ethical concepts and societal values, lack sufficient critical discourse surrounding deployment strategies, and demonstrate inconsistent forms of transparency in their reporting. Based on our findings, we distill provocations and heuristic questions for our community, to recognize the limitations of surveys for meeting the goals of engagement, and to cultivate shared principles to design, deploy, and interpret surveys cautiously and responsibly.
To appear in 7th AAAI Conference on AI, Ethics, and Society (AIES)
Citation data from Semantic Scholar