AI Incidents
Real-world cases where AI systems caused harm, classified by risk domain and entity. Data sourced from 
AIID and classified by 
MIT AI Risk Initiative. CC BY-SA 4.0
244 incidents
University of Toronto's ProctorU Deployment Reportedly Disadvantaged BIPOC Students During Exam Check-Ins
University of Toronto students reported that ProctorU and other online exam-monitoring services created discriminatory and stressful check-in experiences for BIPOC students during remote exams. One U of T student said ProctorU's identity-verification process often failed to recognize her passport, forcing manual checks and consuming exam check-in time. ProctorU said human proctors made final identity and integrity decisions.
Personal voice assistants struggle with black voices, new study shows
A study found that voice recognition tools from Apple, Amazon, Google, IBM, and Microsoft disproportionately made errors when transcribing black speakers.
University of Illinois' Proctorio Remote-Proctoring Deployment Reportedly Raised Student Rights Concerns
Beginning in March 2020, the University of Illinois Urbana-Champaign deployed Proctorio for remote exam proctoring after the COVID-19 shift to online instruction. Students and faculty raised concerns that the software surveilled students' homes and devices, created accessibility barriers, increased testing anxiety, and could flag students when facial-detection tools failed to detect their faces, with particular concern for students of color. UIUC later declined to renew its emergency contract.
Canadian Universities' AI Proctoring Tools Reportedly Collected Student Biometric Data Without Adequate Consent
During the COVID-19 pandemic, Canadian universities reportedly used AI-enabled remote-proctoring tools for online exams. A University of Ottawa project supported by the Office of the Privacy Commissioner of Canada found that tools such as Respondus Monitor, ProctorU, ProctorTrack, Proctorio, ProctorExam, and Examity collected student personal or biometric data under conditions not conducive to meaningful consent.
TikTok's "Suggested Accounts" Algorithm Allegedly Reinforced Racial Bias through Feedback Loops
TikTok's "Suggested Accounts" recommendations allegedly reinforced racial bias despite not basing recommendations on race or creators' profile photo.
Algorithm Used by the Malaysian Judiciary Reportedly Recommended Unusually High Sentencing to a Drug Possession Case
The AI system used by the Malaysian judiciary which explicitly considered age, employment, and socio-economic data provided sentencing to a drug possession case that was alleged by lawyer to be disproportionately high for the crime committed.
Networking Platform Giggle Employs AI to Determine Users’ Gender, Allegedly Excluding Transgender Women
A social networking platform, Giggle, allegedly collected, shared to third-parties, and used sensitive information and biometric data to verify whether a person is a woman via facial recognition, which critics claimed to be discriminatory against women of color and harmful towards trans women.
Detroit Police Allegedly Wrongfully Arrested Black Man Due to Purportedly Faulty Facial Recognition Technology
A Black man was allegedly wrongfully detained by the Detroit Police Department as a result of a purportedly false facial recognition technology (FRT) result.
SN Technologies Reportedly Lied to a New York State School District about Its Facial and Weapon Detection Systems’ Performance
SN Technologies allegedly misled Lockport City Schools about the performance of its AEGIS face and weapons detection systems, downplaying error rates for Black faces and weapon misidentification.
Facebook Feed Algorithms Exposed Low Digitally Skilled Users to More Disturbing Content
Facebook feed algorithms were known by internal research to have harmed people having low digital literacy by exposing them to disturbing content they did not know how to avoid or monitor.
Apple Card's Credit Assessment Algorithm Allegedly Discriminated against Women
Apple Card's credit assessment algorithm was reported by Goldman-Sachs customers to have shown gender bias, in which men received significantly higher credit limits than women with equal credit qualifications.
Job Screening Service Halts Facial Analysis of Applicants
In January 2021, HireVue removed the controversial AI expression tracking tool from its virtual job interview software.
Optum Algorithmic Health Risk Scores Reportedly Underestimated Black Patients' Needs
Researchers analyzing an Optum algorithm used by a large academic hospital reportedly found evidence that it tended to under-predict the health care needs of Black patients, which in turn appeared to result in lower prioritization for extra care programs compared with white patients exhibiting similar health profiles.
Opaque Fraud Detection Algorithm by the UK’s Department of Work and Pensions Allegedly Discriminated against People with Disabilities
People with disabilities were allegedly disproportionately targeted by a benefit fraud detection algorithm which the UK’s Department of Work and Pensions was urged to disclose.
Grab Tweaked Matchmaking Algorithm, Providing Preferential Treatment to Drivers Registered with Affiliated Car Rental Service
Grab Indonesia was fined by the Indonesian Competition Commission (KPPU) for unfairly favoring drivers who rented cars via the Grab-affiliated company Teknologi Pengangkutan Indonesia (TPI), including offering more rides via their matchmaking algorithm.