AI Incidents
Real-world cases where AI systems caused harm, classified by risk domain and entity. Data sourced from 
AIID and classified by 
MIT AI Risk Initiative. CC BY-SA 4.0
197 incidents
iGPT, SimCLR Learned Biased Associations from Internet Training Data
Unsupervised image generation models trained using Internet images such as iGPT and SimCLR were shown to have embedded racial, gender, and intersectional biases, resulting in stereotypical depictions.
Facebook’s Moderation Algorithm Banned Users for Historical Evidence of Slavery
Facebook’s automated content moderation was acknowledged by a company spokesperson to have erroneously censored and banned Australian users from posting an article containing a 1890s photo of Aboriginal men in chains over nudity as historical evidence of slavery in Australia.
Microsoft’s Algorithm Allegedly Selected Photo of the Wrong Mixed-Race Person Featured in a News Story
A news story published on MSN.com featured a photo of the wrong mixed-race person that was allegedly selected by an algorithm, following Microsoft’s layoff and replacement of journalists and editorial workers at its organizations with AI systems.
University of Toronto's ProctorU Deployment Reportedly Disadvantaged BIPOC Students During Exam Check-Ins
University of Toronto students reported that ProctorU and other online exam-monitoring services created discriminatory and stressful check-in experiences for BIPOC students during remote exams. One U of T student said ProctorU's identity-verification process often failed to recognize her passport, forcing manual checks and consuming exam check-in time. ProctorU said human proctors made final identity and integrity decisions.
Personal voice assistants struggle with black voices, new study shows
A study found that voice recognition tools from Apple, Amazon, Google, IBM, and Microsoft disproportionately made errors when transcribing black speakers.
University of Illinois' Proctorio Remote-Proctoring Deployment Reportedly Raised Student Rights Concerns
Beginning in March 2020, the University of Illinois Urbana-Champaign deployed Proctorio for remote exam proctoring after the COVID-19 shift to online instruction. Students and faculty raised concerns that the software surveilled students' homes and devices, created accessibility barriers, increased testing anxiety, and could flag students when facial-detection tools failed to detect their faces, with particular concern for students of color. UIUC later declined to renew its emergency contract.
Canadian Universities' AI Proctoring Tools Reportedly Collected Student Biometric Data Without Adequate Consent
During the COVID-19 pandemic, Canadian universities reportedly used AI-enabled remote-proctoring tools for online exams. A University of Ottawa project supported by the Office of the Privacy Commissioner of Canada found that tools such as Respondus Monitor, ProctorU, ProctorTrack, Proctorio, ProctorExam, and Examity collected student personal or biometric data under conditions not conducive to meaningful consent.
TikTok's "Suggested Accounts" Algorithm Allegedly Reinforced Racial Bias through Feedback Loops
TikTok's "Suggested Accounts" recommendations allegedly reinforced racial bias despite not basing recommendations on race or creators' profile photo.
Detroit Police Allegedly Wrongfully Arrested Black Man Due to Purportedly Faulty Facial Recognition Technology
A Black man was allegedly wrongfully detained by the Detroit Police Department as a result of a purportedly false facial recognition technology (FRT) result.
SN Technologies Reportedly Lied to a New York State School District about Its Facial and Weapon Detection Systems’ Performance
SN Technologies allegedly misled Lockport City Schools about the performance of its AEGIS face and weapons detection systems, downplaying error rates for Black faces and weapon misidentification.
Facebook Feed Algorithms Exposed Low Digitally Skilled Users to More Disturbing Content
Facebook feed algorithms were known by internal research to have harmed people having low digital literacy by exposing them to disturbing content they did not know how to avoid or monitor.
Apple Card's Credit Assessment Algorithm Allegedly Discriminated against Women
Apple Card's credit assessment algorithm was reported by Goldman-Sachs customers to have shown gender bias, in which men received significantly higher credit limits than women with equal credit qualifications.
Optum Algorithmic Health Risk Scores Reportedly Underestimated Black Patients' Needs
Researchers analyzing an Optum algorithm used by a large academic hospital reportedly found evidence that it tended to under-predict the health care needs of Black patients, which in turn appeared to result in lower prioritization for extra care programs compared with white patients exhibiting similar health profiles.
Detroit Police Wrongfully Arrested Black Man Due To Faulty Facial Recognition
A Black man was wrongfully detained by the Detroit Police Department as a result of a false facial recognition (FRT) result.
Amazon’s "Time Off Task" System Made False Assumptions about Workers' Time Management
Amazon’s warehouse worker “time off task" (TOT) tracking system was used to discipline and dismiss workers, falsely assuming workers to have wasted time and failing to account for breaks or equipment issues.