AI Incidents
Real-world cases where AI systems caused harm, classified by risk domain and entity. Data sourced from 
AIID and classified by 
MIT AI Risk Initiative. CC BY-SA 4.0
197 incidents
Korean Chatbot Luda Reportedly Made Offensive Remarks Towards Minority Groups in South Korea
A Korean interactive chatbot was allegedly shown in screenshots to have used derogatory and bigoted language when asked about lesbians, Black people, and people with disabilities.
Online Trolls Allegedly Abused TikTok’s Automated Content Reporting System to Discriminate against Marginalized Creators
TikTok's automated content reporting system was allegedly abused by online trolls to intentionally misreport content created by users of marginalized groups.
Proctoring Algorithm in Online California Bar Exam Flagged an Unusually High Number of Alleged Cheaters
The proctoring algorithm used in a California bar exam cited a third of thousands of applicants as cheaters, resulting in allegations where exam takers were instructed to prove otherwise without seeing their incriminating video evidence.
Court Rules Deliveroo Used 'Discriminatory' Algorithm
In December 2020, an Italian court ruled that Deliveroo’s employee ‘reliability’ algorithm illegally discriminated against workers with legitimate reasons for cancelling shifts.
AI Spam Filters Allegedly Block Legitimate Emails Based on Biased Keyword Detection
AlgorithmWatch tested spam filtering algorithms across Gmail, Yahoo, Outlook, GMX, and LaPoste. Their findings reportedly showed that Microsoft Outlook’s spam filter flagged emails based on specific keywords that led to racial and content-based biases blocking legitimate communications. Emails mentioning Nigeria or containing certain financial and sexual health terms were found to be disproportionately marked as spam.
Researchers find evidence of racial, gender, and socioeconomic bias in chest X-ray classifiers
A study by the University of Toronto, the Vector Institute, and MIT showed the input databases that trained AI systems used to classify chest X-rays led the systems to show gender, socioeconomic, and racial biases.
UK passport photo checker shows bias against dark-skinned women
UK passport photo checker shows bias against dark-skinned women.
Twitter’s Image Cropping Tool Allegedly Showed Gender and Racial Bias
Twitter's photo cropping algorithm was revealed by researchers to favor white and women faces in photos containing multiple faces, prompting the company to stop its use on mobile platform.
Philosophy AI Tentatively Produced Offensive Results for Certain Prompts
Philosopher AI as built on top of GPT-3 was reported by its users for having strong tendencies to produce offensive results when given prompts on certain topics such as feminism and Ethiopia.
OpenAI's GPT-3 Associated Muslims with Violence
Users and researchers revealed generative AI GPT-3 associating Muslims to violence in prompts, resulting in disturbingly racist and explicit outputs such as casting Muslim actor as a terrorist.
Bug in Instagram’s “Related Hashtags” Algorithm Allegedly Caused Disproportionate Treatment of Political Hashtags
A bug was reported by Instagram’s spokesperson to have prevented an algorithm from populating related hashtags for thousands of hashtags, resulting in an allege preferential treatment for some politically partisan hashtags.
Genderify’s AI to Predict a Person’s Gender Revealed by Free API Users to Exhibit Bias
A company's AI predicting a person's gender based on their name, email address, or username was reported by its users to show biased and inaccurate results.
Thoughts App Allegedly Created Toxic Tweets
Tweets created by Thoughts, a tweet generation app that leverages OpenAI’s GPT-3, allegedly exhibited toxicity when given prompts related to minority groups.
Facebook's AI Put "Primates" Label on Video Featuring Black Men
Facebook's AI mislabeled video featuring Black men as a video about "primates," resulting in an offensive prompt message for users who watched the video.
Image Upscaling Algorithm PULSE Allegedly Produced Facial Images with Caucasian Features More Often
Image upscaling tool PULSE powered by NVIDIA's StyleGAN reportedly generated faces with Caucasian features more often, although AI academics, engineers, and researchers were not in agreement about where the source of bias was.