AI Incidents
Real-world cases where AI systems caused harm, classified by risk domain and entity. Data sourced from 
AIID and classified by 
MIT AI Risk Initiative. CC BY-SA 4.0
197 incidents
Chatbot for DPD Malfunctioned and Swore at Customers and Criticized Its Own Company
DPD's AI chatbot, used for customer service, appeared to malfunction following a system update, leading to inappropriate responses including swearing and criticizing the company. The incident, which became viral on social media, occurred after the chatbot was updated, prompting DPD to disable the malfunctioning AI component.
Rite Aid Facial Recognition Disproportionately Misidentified Minority Shoppers as Shoplifters
Rite Aid used facial recognition technology from October 2012 to July 2020, allegedly leading to disproportionate misidentifications of women, Black, Latino, and Asian shoppers as "likely" shoplifters. The FTC settlement prohibits Rite Aid from using this technology in stores for five years.
Child Sexual Abuse Material Taints Image Generators
The LAION-5B dataset (a commonly used dataset with more than 5 billion image-description pairs) was found by researchers to contain child sexual abuse material (CSAM), which increases the likelihood that downstream models will produce CSAM imagery. The discovery taints models built with the LAION dataset requiring many organizations to retrain those models. Additionally, LAION must now scrub the dataset of the imagery.
Navy Federal Credit Union Faces Allegations of Racial Bias in Mortgage Approvals
Navy Federal Credit Union, serving military members and veterans, faced allegations of racial bias in its mortgage approval process, which relies on automated underwriting technology. In 2022, data revealed significant disparities in loan approvals, with over 50% of Black applicants denied, compared to higher approval rates for white applicants.
Microsoft AI Is Alleged to Have Generated Violent Imagery of Minorities and Public Figures
Microsoft's AI Image Creator, integrated with Bing and Windows Paint, allegedly produced disturbingly violent and graphic images featuring members of minority groups and public figures like Joe Biden and Pope Francis.
Facebook Messenger AI Stickers Generate Ethical and Content Moderation Concerns
Facebook Messenger AI stickers, a feature by Meta, allows users to generate personalized stickers via AI for use in conversations. While the feature has been praised for its creativity, it has also stirred controversy for its alleged production of inappropriate or offensive content. This has raised questions about the effectiveness of Meta's content moderation measures and the ethical responsibilities associated with AI-driven content generation.
Peppermill Casino Facial Recognition System Reportedly Misidentified Individual, Leading to Wrongful Arrest in Reno
Purported facial recognition technology deployed at Reno's Peppermill Casino reportedly misidentified Jason Killinger as an individual previously banned from the venue. Casino security reportedly detained Killinger and contacted police, leading to his arrest by the Reno PD. He was reportedly held for 11 hours, including several hours in handcuffs, before fingerprinting reportedly confirmed the misidentification. The incident resulted in alleged physical injury and civil liberties harms.
AI Photo Filter Lightens Skin, Changes Eye Color in Student's 'Professional' Image
An AI application modified an MIT student's photo to appear 'professional' by lightening her skin and changing her eye color to blue, highlighting the racial bias in the training data of the program.
Harmful Stereotyping of Non-Cisgendered People via Text-to-Image Systems
Text-to-image systems such as DALL-E are allegedly generating biased and often insulting representations of non-cisgender identities. The systems tend to generate stereotypical and sexualized images when prompted with gender identity terms like "trans," "nonbinary," or "queer," highlighting systemic issues of bias.
Alleged Gender Discrimination in Facebook Job Ads Algorithm
Facebook's ad delivery algorithm allegedly disproportionately showed job advertisements to one gender. Despite claims of non-discrimination, the algorithm's actions seem to perpetuate societal biases, which in turn could potentially limit opportunities for certain groups and hinder gender equity in the workplace.
Instagram Algorithms Allegedly Promote Accounts Facilitating Child Sex Abuse Content
An investigation disclosed that Instagram's recommendation algorithms are promoting accounts that facilitate and sell child sexual abuse material (CSAM). The study, conducted by The Wall Street Journal and researchers at Stanford University and the University of Massachusetts Amherst, indicates that Instagram's algorithms not only allow for the discovery of such accounts through keyword searches but also actively recommend them to users within the network. The issue is especially concerning given Instagram's popularity among teenagers.
Racial Bias in Lung Function Diagnostic Algorithm Leads to Underdiagnosis in Black Men
A study published in JAMA Network Open reveals that racial bias built into a commonly used medical diagnostic algorithm for lung function may be leading to underdiagnoses of breathing problems in Black men. The study suggests that as many as 40% more Black male patients might have been accurately diagnosed if the software were not racially biased. The software algorithm adjusts diagnostic thresholds based on race, affecting medical treatments and interventions.
Apparent Failure to Accurately Label Primates in Image Recognition Software Due to Alleged Fear of Racial Bias
Eight years after Google Photos mislabeled images of Black individuals as "gorillas," image recognition software by Google, Apple, Amazon, and Microsoft still shows signs of either avoiding or inaccurately categorizing primates. Tests reveal that Google and Apple Photos refrain from labeling primates altogether, possibly to avoid the risk of perpetuating racial stereotypes. Microsoft OneDrive fails to identify any animals, while Amazon Photos overgeneralizes in its labeling.
Australian Terrorism Prediction Tool Reportedly Disparately Impacts Persons with Autism
An independent report found that the Vera-2R tool, designed to predict the risk of future terrorist activities, considered autism as a risk factor despite lacking empirical evidence to support this claim. The report called into question the tool's overall validity and reliability, stating it was "extremely poor" at accurately predicting risk. The inclusion of autism as a risk factor had potentially serious implications for the tool's use and credibility.
Snapchat's My AI Reported for Lacking Protection for Children
Snapchat's ChatGPT-powered My AI was reported for lacking safeguards for children, such as telling a user who tested the chatbot by pretending to sign up as a 13-year-old girl to lie to her parents about having a romantic getaway with an older man, and sharing tips on how to cover up evidence of abuse.