AI Incidents
Real-world cases where AI systems caused harm, classified by risk domain and entity. Data sourced from 
AIID and classified by 
MIT AI Risk Initiative. CC BY-SA 4.0
244 incidents
Honorlock's AI Proctoring System Reportedly Led Broward College to Accuse a Teenager of Cheating
During a February 2022 online biology exam, Honorlock reportedly flagged a Florida teenager taking a Broward College class as suspicious. The student denied cheating, but Broward College allegedly relied on the remote-proctoring system's flag in pursuing an academic-dishonesty accusation that the student said was unfounded.
Justice Department’s Recidivism Risk Algorithm PATTERN Allegedly Caused Persistent Disparities Along Racial Lines
Department of Justice’s inmate-recidivism risk assessment tool was reported to have produced racially uneven results, misclassifying risk levels for inmates of color.
Replika's "AI Companions" Reportedly Abused by Its Users
Replika's AI-powered "digital companions" was allegedly abused by their users, who posted on Reddit abusive behaviors and interactions such as using slurs, roleplaying violent acts, and stimulating sexual abuse.
TikTok's "For You" Algorithm Allegedly Abused by Online Personality to Promote Anti-Women Hate
TikTok’s “For You” algorithm allegedly boosted or was manipulated by an online personality to artificially boost his content which promotes extreme misogynistic views towards teenagers and men, despite breaking its rules.
Alexa Recommended Dangerous TikTok Challenge to Ten-Year-Old Girl
Amazon’s voice assistant Alexa suggested “the penny challenge,” which involves dangerously touching a coin to the prongs of a half-exposed plug, when a ten-year-old girl asked for a challenge to do.
UK Government AI Allegedly Targets Disproportionate Numbers of Certain Nationals for Fraud Review
The UK's Department for Work and Pensions (DWP) faced scrutiny after many Bulgarian nationals reported unexplained suspensions of their Universal Credit benefits. The MP for Edmonton raised concerns about potential nationality-based targeting for benefit fraud investigations, leading to poverty and homelessness among affected individuals. The Home Office's own equality impact assessment found it was flagging a disproportionate number of marriages from Greece, Albania, Bulgaria and Romania.
Facebook’s Hate Speech Detection Algorithms Allegedly Disproportionately Failed to Remove Racist Content towards Minority Groups
Facebook’s hate-speech detection algorithms was found by company researchers to have under-reported less common but more harmful content that was more often experienced by minority groups such as Black, Muslim, LGBTQ, and Jewish users.
Research Prototype AI, Delphi, Reportedly Gave Racially Biased Answers on Ethics
A publicly accessible research model that was trained via Reddit threads showed racially biased advice on moral dilemmas, allegedly demonstrating limitations of language-based models trained on moral judgments.
Startup's Accent Translation AI Denounced as Reinforcing Racial Bias
A startup’s use of AI voice technology to alter or remove accents for call center agents was scrutinized by critics as reaffirming bias, despite the company’s claim.
Facebook, Instagram, and Twitter Failed to Proactively Remove Targeted Racist Remarks via Automated Systems
Facebook's, Instagram's, and Twitter's automated content moderation failed to proactively remove racist remarks and posts directing at Black football players after finals loss, allegedly largely relying on user reports of harassment.
Skating Rink’s Facial Recognition Cameras Misidentified Black Teenager as Banned Troublemaker
A Black teenager living in Livonia, Michigan was incorrectly stopped from entering a roller skating rink after its facial-recognition cameras misidentified her as another person who had been previously banned for starting a skirmish with other skaters.
State Farm Allegedly Discriminated against Black Customers in Claim Payout
State Farm's automated claims processing method was alleged in a class action lawsuit to have disproportionately against Black policyholders when paying out insurance claims.
Kannada Insulted by Google's Featured Answer as "Ugliest Language in India"
Google's knowledge-graph-powered algorithm showed Kannada in its featured Answer Box when prompted "ugliest language in India," causing outrage from Kannada-speaking people and government.
Facebook, Instagram, and Twitter Cited Errors in Automated Systems as Cause for Blocking pro-Palestinian Content on Israeli-Palestinian Conflict
Facebook, Instagram, and Twitter wrongly blocked or restricted millions of pro-Palestinian posts and accounts related to the Israeli-Palestinian conflict, citing errors in their automated content moderation system.
Replika's AI Partners Reportedly Sexually Harassed Users
Replika's "AI companions" were reported by users for sexually harassing them, such as sending unwanted sexual messages or behaving aggressively.