AI Incidents
Real-world cases where AI systems caused harm, classified by risk domain and entity. Data sourced from 
AIID and classified by 
MIT AI Risk Initiative. CC BY-SA 4.0
217 incidents
Bug in Instagram’s “Related Hashtags” Algorithm Allegedly Caused Disproportionate Treatment of Political Hashtags
A bug was reported by Instagram’s spokesperson to have prevented an algorithm from populating related hashtags for thousands of hashtags, resulting in an allege preferential treatment for some politically partisan hashtags.
Genderify’s AI to Predict a Person’s Gender Revealed by Free API Users to Exhibit Bias
A company's AI predicting a person's gender based on their name, email address, or username was reported by its users to show biased and inaccurate results.
Thoughts App Allegedly Created Toxic Tweets
Tweets created by Thoughts, a tweet generation app that leverages OpenAI’s GPT-3, allegedly exhibited toxicity when given prompts related to minority groups.
Facebook's AI Put "Primates" Label on Video Featuring Black Men
Facebook's AI mislabeled video featuring Black men as a video about "primates," resulting in an offensive prompt message for users who watched the video.
Image Upscaling Algorithm PULSE Allegedly Produced Facial Images with Caucasian Features More Often
Image upscaling tool PULSE powered by NVIDIA's StyleGAN reportedly generated faces with Caucasian features more often, although AI academics, engineers, and researchers were not in agreement about where the source of bias was.
iGPT, SimCLR Learned Biased Associations from Internet Training Data
Unsupervised image generation models trained using Internet images such as iGPT and SimCLR were shown to have embedded racial, gender, and intersectional biases, resulting in stereotypical depictions.
Facebook’s Moderation Algorithm Banned Users for Historical Evidence of Slavery
Facebook’s automated content moderation was acknowledged by a company spokesperson to have erroneously censored and banned Australian users from posting an article containing a 1890s photo of Aboriginal men in chains over nudity as historical evidence of slavery in Australia.
Microsoft’s Algorithm Allegedly Selected Photo of the Wrong Mixed-Race Person Featured in a News Story
A news story published on MSN.com featured a photo of the wrong mixed-race person that was allegedly selected by an algorithm, following Microsoft’s layoff and replacement of journalists and editorial workers at its organizations with AI systems.
University of Toronto's ProctorU Deployment Reportedly Disadvantaged BIPOC Students During Exam Check-Ins
University of Toronto students reported that ProctorU and other online exam-monitoring services created discriminatory and stressful check-in experiences for BIPOC students during remote exams. One U of T student said ProctorU's identity-verification process often failed to recognize her passport, forcing manual checks and consuming exam check-in time. ProctorU said human proctors made final identity and integrity decisions.
Personal voice assistants struggle with black voices, new study shows
A study found that voice recognition tools from Apple, Amazon, Google, IBM, and Microsoft disproportionately made errors when transcribing black speakers.
University of Illinois' Proctorio Remote-Proctoring Deployment Reportedly Raised Student Rights Concerns
Beginning in March 2020, the University of Illinois Urbana-Champaign deployed Proctorio for remote exam proctoring after the COVID-19 shift to online instruction. Students and faculty raised concerns that the software surveilled students' homes and devices, created accessibility barriers, increased testing anxiety, and could flag students when facial-detection tools failed to detect their faces, with particular concern for students of color. UIUC later declined to renew its emergency contract.
Canadian Universities' AI Proctoring Tools Reportedly Collected Student Biometric Data Without Adequate Consent
During the COVID-19 pandemic, Canadian universities reportedly used AI-enabled remote-proctoring tools for online exams. A University of Ottawa project supported by the Office of the Privacy Commissioner of Canada found that tools such as Respondus Monitor, ProctorU, ProctorTrack, Proctorio, ProctorExam, and Examity collected student personal or biometric data under conditions not conducive to meaningful consent.
TikTok's "Suggested Accounts" Algorithm Allegedly Reinforced Racial Bias through Feedback Loops
TikTok's "Suggested Accounts" recommendations allegedly reinforced racial bias despite not basing recommendations on race or creators' profile photo.
Algorithm Used by the Malaysian Judiciary Reportedly Recommended Unusually High Sentencing to a Drug Possession Case
The AI system used by the Malaysian judiciary which explicitly considered age, employment, and socio-economic data provided sentencing to a drug possession case that was alleged by lawyer to be disproportionately high for the crime committed.
Networking Platform Giggle Employs AI to Determine Users’ Gender, Allegedly Excluding Transgender Women
A social networking platform, Giggle, allegedly collected, shared to third-parties, and used sensitive information and biometric data to verify whether a person is a woman via facial recognition, which critics claimed to be discriminatory against women of color and harmful towards trans women.